Note: This is test shard 7 of 8.
[==========] Running 28 tests from 8 test suites.
[----------] Global test environment set-up.
[----------] 20 tests from ClientTest
[ RUN ] ClientTest.ConnectToClusterNoMasterAddressSpecified
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250904 07:52:53.139469 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36649
I20250904 07:52:53.141247 413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 07:52:53.142931 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:53.153798 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.153831 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.154440 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:53.155239 413 server_base.cc:1047] running on GCE node
I20250904 07:52:53.156587 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:53.156740 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:53.156802 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972373156789 us; error 0 us; skew 500 ppm
I20250904 07:52:53.157121 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:53.159580 413 webserver.cc:480] Webserver started at http://127.0.103.126:35827/ using document root <none> and password file <none>
I20250904 07:52:53.160172 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:53.160254 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:53.160575 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:53.163487 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "56c7a49bc1bb4fb3b08cad42e264113e"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.168908 413 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20250904 07:52:53.171746 426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.173199 413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250904 07:52:53.173354 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "56c7a49bc1bb4fb3b08cad42e264113e"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.173513 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:53.198927 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:53.199671 413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 07:52:53.199940 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:53.223692 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36649
I20250904 07:52:53.223696 487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36649 every 8 connection(s)
I20250904 07:52:53.227232 488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.239248 488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: Bootstrap starting.
I20250904 07:52:53.242769 488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.243809 488 log.cc:826] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: Log is configured to *not* fsync() on all Append() calls
I20250904 07:52:53.246316 488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: No bootstrap required, opened a new log
I20250904 07:52:53.252154 488 raft_consensus.cc:357] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER }
I20250904 07:52:53.252406 488 raft_consensus.cc:383] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.252499 488 raft_consensus.cc:738] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c7a49bc1bb4fb3b08cad42e264113e, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.253194 488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER }
I20250904 07:52:53.253381 488 raft_consensus.cc:397] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.253474 488 raft_consensus.cc:491] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.253603 488 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.255445 488 raft_consensus.cc:513] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER }
I20250904 07:52:53.255923 488 leader_election.cc:304] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c7a49bc1bb4fb3b08cad42e264113e; no voters:
I20250904 07:52:53.256448 488 leader_election.cc:290] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.256582 491 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.257302 491 raft_consensus.cc:695] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 1 LEADER]: Becoming Leader. State: Replica: 56c7a49bc1bb4fb3b08cad42e264113e, State: Running, Role: LEADER
I20250904 07:52:53.257822 491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER }
I20250904 07:52:53.258431 488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:53.261369 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER } }
I20250904 07:52:53.261720 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.262250 493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c7a49bc1bb4fb3b08cad42e264113e. Latest consensus state: current_term: 1 leader_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c7a49bc1bb4fb3b08cad42e264113e" member_type: VOTER } }
I20250904 07:52:53.262529 493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.264061 501 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:53.268978 501 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:53.269529 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:53.276983 501 catalog_manager.cc:1349] Generated new cluster ID: 6067c82a0f2c435aaaec3bc9191f2051
I20250904 07:52:53.277086 501 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:53.310564 501 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:53.311770 501 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:53.323203 501 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: Generated new TSK 0
I20250904 07:52:53.324031 501 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:53.334599 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:53.338709 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.339291 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.339792 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:53.340279 413 server_base.cc:1047] running on GCE node
I20250904 07:52:53.340534 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:53.340623 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:53.340685 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972373340662 us; error 0 us; skew 500 ppm
I20250904 07:52:53.340912 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:53.342154 413 webserver.cc:480] Webserver started at http://127.0.103.65:46613/ using document root <none> and password file <none>
I20250904 07:52:53.342383 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:53.342468 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:53.342620 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:53.343269 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.345966 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250904 07:52:53.347698 518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.348060 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:53.348173 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.348299 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.ConnectToClusterNoMasterAddressSpecified.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:53.358003 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:53.358563 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:53.361377 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:53.361483 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.361630 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:53.361696 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.400466 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:46289
I20250904 07:52:53.400487 586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:46289 every 8 connection(s)
I20250904 07:52:53.404798 587 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36649
I20250904 07:52:53.405138 587 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:53.406061 587 heartbeater.cc:507] Master 127.0.103.126:36649 requested a full tablet report, sending...
I20250904 07:52:53.408558 443 ts_manager.cc:194] Registered new tserver with Master: d05d1ad23fcf4d8f9db5eefc8dd70cf3 (127.0.103.65:46289)
I20250904 07:52:53.409829 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.007015812s
I20250904 07:52:53.411397 443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51446
I20250904 07:52:53.427620 587 heartbeater.cc:499] Master 127.0.103.126:36649 was elected leader, sending a full tablet report...
I20250904 07:52:53.425004 443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51456:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:53.454823 548 tablet_service.cc:1468] Processing CreateTablet for tablet f115f62fe18e422da4742e77d79a515c (DEFAULT_TABLE table=client-testtb [id=ef31c8a58aac4892aa0f4f490d163376]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:53.454828 549 tablet_service.cc:1468] Processing CreateTablet for tablet 36075cc936b84244aa6c4f5591c15b06 (DEFAULT_TABLE table=client-testtb [id=ef31c8a58aac4892aa0f4f490d163376]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:53.455682 548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f115f62fe18e422da4742e77d79a515c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.455992 549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36075cc936b84244aa6c4f5591c15b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.462203 599 tablet_bootstrap.cc:492] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Bootstrap starting.
I20250904 07:52:53.464118 599 tablet_bootstrap.cc:654] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.466789 599 tablet_bootstrap.cc:492] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: No bootstrap required, opened a new log
I20250904 07:52:53.466934 599 ts_tablet_manager.cc:1397] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250904 07:52:53.467566 599 raft_consensus.cc:357] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.467734 599 raft_consensus.cc:383] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.467850 599 raft_consensus.cc:738] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05d1ad23fcf4d8f9db5eefc8dd70cf3, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.468067 599 consensus_queue.cc:260] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.468218 599 raft_consensus.cc:397] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.468323 599 raft_consensus.cc:491] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.468447 599 raft_consensus.cc:3058] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.470353 599 raft_consensus.cc:513] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.470561 599 leader_election.cc:304] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05d1ad23fcf4d8f9db5eefc8dd70cf3; no voters:
I20250904 07:52:53.471024 599 leader_election.cc:290] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.471189 601 raft_consensus.cc:2802] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.471873 599 ts_tablet_manager.cc:1428] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20250904 07:52:53.472247 599 tablet_bootstrap.cc:492] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Bootstrap starting.
I20250904 07:52:53.474023 599 tablet_bootstrap.cc:654] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.475158 601 raft_consensus.cc:695] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 LEADER]: Becoming Leader. State: Replica: d05d1ad23fcf4d8f9db5eefc8dd70cf3, State: Running, Role: LEADER
I20250904 07:52:53.475533 601 consensus_queue.cc:237] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.475839 599 tablet_bootstrap.cc:492] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: No bootstrap required, opened a new log
I20250904 07:52:53.475987 599 ts_tablet_manager.cc:1397] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:53.476728 599 raft_consensus.cc:357] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.476974 599 raft_consensus.cc:383] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.477092 599 raft_consensus.cc:738] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05d1ad23fcf4d8f9db5eefc8dd70cf3, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.477442 599 consensus_queue.cc:260] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.477653 599 raft_consensus.cc:397] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.477783 599 raft_consensus.cc:491] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.477927 599 raft_consensus.cc:3058] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.479310 443 catalog_manager.cc:5582] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d05d1ad23fcf4d8f9db5eefc8dd70cf3 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:53.480734 599 raft_consensus.cc:513] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.481000 599 leader_election.cc:304] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05d1ad23fcf4d8f9db5eefc8dd70cf3; no voters:
I20250904 07:52:53.481230 599 leader_election.cc:290] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.482661 602 raft_consensus.cc:2802] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.485347 599 ts_tablet_manager.cc:1428] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20250904 07:52:53.485893 602 raft_consensus.cc:695] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 LEADER]: Becoming Leader. State: Replica: d05d1ad23fcf4d8f9db5eefc8dd70cf3, State: Running, Role: LEADER
I20250904 07:52:53.486282 602 consensus_queue.cc:237] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } }
I20250904 07:52:53.489406 443 catalog_manager.cc:5582] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d05d1ad23fcf4d8f9db5eefc8dd70cf3 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05d1ad23fcf4d8f9db5eefc8dd70cf3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46289 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:53.499976 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:53.511286 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:53.511731 413 tablet_replica.cc:331] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3: stopping tablet replica
I20250904 07:52:53.512125 413 raft_consensus.cc:2241] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.512362 413 raft_consensus.cc:2270] T 36075cc936b84244aa6c4f5591c15b06 P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.513835 413 tablet_replica.cc:331] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3: stopping tablet replica
I20250904 07:52:53.513975 413 raft_consensus.cc:2241] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.514127 413 raft_consensus.cc:2270] T f115f62fe18e422da4742e77d79a515c P d05d1ad23fcf4d8f9db5eefc8dd70cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.529351 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:53.535097 413 master.cc:561] Master@127.0.103.126:36649 shutting down...
I20250904 07:52:53.545640 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.545903 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.546141 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 56c7a49bc1bb4fb3b08cad42e264113e: stopping tablet replica
I20250904 07:52:53.560783 413 master.cc:583] Master@127.0.103.126:36649 shutdown complete.
[ OK ] ClientTest.ConnectToClusterNoMasterAddressSpecified (438 ms)
[ RUN ] ClientTest.TestScanAtFutureTimestamp
I20250904 07:52:53.575208 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:33915
I20250904 07:52:53.575842 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:53.578388 614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.578598 613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.579842 616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:53.580209 413 server_base.cc:1047] running on GCE node
I20250904 07:52:53.580519 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:53.580622 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:53.580688 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972373580667 us; error 0 us; skew 500 ppm
I20250904 07:52:53.580904 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:53.582190 413 webserver.cc:480] Webserver started at http://127.0.103.126:45937/ using document root <none> and password file <none>
I20250904 07:52:53.582463 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:53.582551 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:53.582710 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:53.583532 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "8b881c573bd9441c9e53b48c27ee9ab9"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.586812 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250904 07:52:53.588831 621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.589257 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:53.589385 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "8b881c573bd9441c9e53b48c27ee9ab9"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.589515 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:53.617025 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:53.617650 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:53.639380 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:33915
I20250904 07:52:53.639405 682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:33915 every 8 connection(s)
I20250904 07:52:53.641009 683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.647904 683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9: Bootstrap starting.
I20250904 07:52:53.649942 683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.652433 683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9: No bootstrap required, opened a new log
I20250904 07:52:53.653384 683 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER }
I20250904 07:52:53.653577 683 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.653689 683 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b881c573bd9441c9e53b48c27ee9ab9, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.653944 683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER }
I20250904 07:52:53.654134 683 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.654232 683 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.654350 683 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.657187 683 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER }
I20250904 07:52:53.657449 683 leader_election.cc:304] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b881c573bd9441c9e53b48c27ee9ab9; no voters:
I20250904 07:52:53.657980 683 leader_election.cc:290] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.658110 686 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.658963 683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:53.664757 686 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 8b881c573bd9441c9e53b48c27ee9ab9, State: Running, Role: LEADER
I20250904 07:52:53.665099 686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER }
I20250904 07:52:53.666083 687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER } }
I20250904 07:52:53.666288 687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.666749 688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b881c573bd9441c9e53b48c27ee9ab9. Latest consensus state: current_term: 1 leader_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b881c573bd9441c9e53b48c27ee9ab9" member_type: VOTER } }
I20250904 07:52:53.666970 688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.674118 698 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:53.676563 698 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:53.676949 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:53.679754 698 catalog_manager.cc:1349] Generated new cluster ID: 101f08f9c4004e66b75be30f8b488889
I20250904 07:52:53.679849 698 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:53.706899 698 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:53.708024 698 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:53.719491 698 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9: Generated new TSK 0
I20250904 07:52:53.719754 698 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:53.741912 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:53.745723 705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.746542 706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.747856 708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:53.748008 413 server_base.cc:1047] running on GCE node
I20250904 07:52:53.748435 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:53.748515 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:53.748600 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972373748560 us; error 0 us; skew 500 ppm
I20250904 07:52:53.748764 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:53.750061 413 webserver.cc:480] Webserver started at http://127.0.103.65:43595/ using document root <none> and password file <none>
I20250904 07:52:53.750280 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:53.750380 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:53.750507 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:53.751175 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "8cd05c9f848642948a9f6301b8894676"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.753788 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:53.755414 713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.755753 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:53.755852 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "8cd05c9f848642948a9f6301b8894676"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.755986 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScanAtFutureTimestamp.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:53.762202 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:53.763284 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:53.764866 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:53.764989 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.765127 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:53.765241 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.796057 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38455
I20250904 07:52:53.796093 783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38455 every 8 connection(s)
I20250904 07:52:53.797760 784 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33915
I20250904 07:52:53.797866 784 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:53.798254 784 heartbeater.cc:507] Master 127.0.103.126:33915 requested a full tablet report, sending...
I20250904 07:52:53.798996 638 ts_manager.cc:194] Registered new tserver with Master: 8cd05c9f848642948a9f6301b8894676 (127.0.103.65:38455)
I20250904 07:52:53.799388 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002298765s
I20250904 07:52:53.800009 638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42798
I20250904 07:52:53.810225 638 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42806:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:53.811206 784 heartbeater.cc:499] Master 127.0.103.126:33915 was elected leader, sending a full tablet report...
I20250904 07:52:53.824156 747 tablet_service.cc:1468] Processing CreateTablet for tablet 57539e1a364846ee9809e390a11450bc (DEFAULT_TABLE table=client-testtb [id=a3a5b6cef5284387bd503a2c43bebfd1]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:53.824136 748 tablet_service.cc:1468] Processing CreateTablet for tablet 8119b9d59d044c0e93289a5d413e0955 (DEFAULT_TABLE table=client-testtb [id=a3a5b6cef5284387bd503a2c43bebfd1]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:53.824751 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8119b9d59d044c0e93289a5d413e0955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.824996 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57539e1a364846ee9809e390a11450bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.831800 794 tablet_bootstrap.cc:492] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: Bootstrap starting.
I20250904 07:52:53.833771 794 tablet_bootstrap.cc:654] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.835866 794 tablet_bootstrap.cc:492] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: No bootstrap required, opened a new log
I20250904 07:52:53.835980 794 ts_tablet_manager.cc:1397] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:53.836544 794 raft_consensus.cc:357] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.836681 794 raft_consensus.cc:383] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.836774 794 raft_consensus.cc:738] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd05c9f848642948a9f6301b8894676, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.836978 794 consensus_queue.cc:260] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.837114 794 raft_consensus.cc:397] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.837184 794 raft_consensus.cc:491] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.837291 794 raft_consensus.cc:3058] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.838886 794 raft_consensus.cc:513] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.839083 794 leader_election.cc:304] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd05c9f848642948a9f6301b8894676; no voters:
I20250904 07:52:53.839475 794 leader_election.cc:290] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.839614 796 raft_consensus.cc:2802] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.840229 794 ts_tablet_manager.cc:1428] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:52:53.840224 796 raft_consensus.cc:695] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd05c9f848642948a9f6301b8894676, State: Running, Role: LEADER
I20250904 07:52:53.840536 794 tablet_bootstrap.cc:492] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: Bootstrap starting.
I20250904 07:52:53.840562 796 consensus_queue.cc:237] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.842528 638 catalog_manager.cc:5582] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cd05c9f848642948a9f6301b8894676 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "8cd05c9f848642948a9f6301b8894676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:53.842851 794 tablet_bootstrap.cc:654] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.844646 794 tablet_bootstrap.cc:492] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: No bootstrap required, opened a new log
I20250904 07:52:53.844801 794 ts_tablet_manager.cc:1397] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:53.845434 794 raft_consensus.cc:357] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.845597 794 raft_consensus.cc:383] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.845674 794 raft_consensus.cc:738] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd05c9f848642948a9f6301b8894676, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.845957 794 consensus_queue.cc:260] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.846123 794 raft_consensus.cc:397] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.846228 794 raft_consensus.cc:491] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.846335 794 raft_consensus.cc:3058] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.848230 794 raft_consensus.cc:513] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.848415 794 leader_election.cc:304] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd05c9f848642948a9f6301b8894676; no voters:
I20250904 07:52:53.848601 794 leader_election.cc:290] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.848654 796 raft_consensus.cc:2802] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.848829 796 raft_consensus.cc:695] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd05c9f848642948a9f6301b8894676, State: Running, Role: LEADER
I20250904 07:52:53.849077 796 consensus_queue.cc:237] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } }
I20250904 07:52:53.849153 794 ts_tablet_manager.cc:1428] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:52:53.850804 638 catalog_manager.cc:5582] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cd05c9f848642948a9f6301b8894676 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "8cd05c9f848642948a9f6301b8894676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd05c9f848642948a9f6301b8894676" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38455 } health_report { overall_health: HEALTHY } } }
W20250904 07:52:53.874327 734 tablet_service.cc:2974] error setting up scanner 79703bb67dc544be8077d5a0ac283b84 with request: Invalid argument: cannot verify timestamp: Snapshot time P: 1756972433858416 usec, L: 0 in the future. Max allowed timestamp is P: 1756972383874208 usec, L: 0: new_scan_request { tablet_id: "8119b9d59d044c0e93289a5d413e0955" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } projected_columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" write_default_value: "90\000\000" immutable: false } read_mode: READ_AT_SNAPSHOT snap_timestamp: 7196559089084071936 propagated_timestamp: 7196558843378978816 cache_blocks: true order_mode: UNORDERED row_format_flags: 0 authz_token { token_data: "\010\201\215\345\305\006\"3\n\005slave\022*\n a3a5b6cef5284387bd503a2c43bebfd1\020\001\030\001 \001(\001" signature: "&\362\201e\362\310\363\222\322\322`Q\247\2044\273j\364\304\241\225\373RL\364]s\351K\004\354\027f\330\023\351\014\255\302\2758\352\272\316\344\232\"e\035\037\251\314G\265-\020\016\353\371Utp\000\276""&\362\201e\362\310\363\222\322\322`Q\247\2044\273j\364\304\241\225\373RL\364]s\351K\004\354\027f\330\023\351\014\255\302\2758\352\272\316\344\232\"e\035\037\251\314G\265-\020\016\353\371Utp\000\276" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d4ad8d60124a40c58158355e16aebbcb"
I20250904 07:52:53.877467 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:53.886863 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:53.887214 413 tablet_replica.cc:331] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676: stopping tablet replica
I20250904 07:52:53.887439 413 raft_consensus.cc:2241] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.887617 413 raft_consensus.cc:2270] T 57539e1a364846ee9809e390a11450bc P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.888466 413 tablet_replica.cc:331] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676: stopping tablet replica
I20250904 07:52:53.888602 413 raft_consensus.cc:2241] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.888772 413 raft_consensus.cc:2270] T 8119b9d59d044c0e93289a5d413e0955 P 8cd05c9f848642948a9f6301b8894676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.893597 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:53.898849 413 master.cc:561] Master@127.0.103.126:33915 shutting down...
I20250904 07:52:53.909409 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:53.909613 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:53.909794 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8b881c573bd9441c9e53b48c27ee9ab9: stopping tablet replica
I20250904 07:52:53.923828 413 master.cc:583] Master@127.0.103.126:33915 shutdown complete.
[ OK ] ClientTest.TestScanAtFutureTimestamp (358 ms)
[ RUN ] ClientTest.TestInvalidPredicates
I20250904 07:52:53.933101 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:43771
I20250904 07:52:53.933597 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:53.935863 806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.936103 805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:53.937073 808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:53.937110 413 server_base.cc:1047] running on GCE node
I20250904 07:52:53.937543 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:53.937618 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:53.937700 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972373937662 us; error 0 us; skew 500 ppm
I20250904 07:52:53.937862 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:53.939011 413 webserver.cc:480] Webserver started at http://127.0.103.126:42769/ using document root <none> and password file <none>
I20250904 07:52:53.939220 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:53.939320 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:53.939450 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:53.940062 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "a0dac51cffb94d80be7fb582eaeb48f4"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.942482 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:53.944084 813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:53.944453 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:53.944556 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "a0dac51cffb94d80be7fb582eaeb48f4"
format_stamp: "Formatted at 2025-09-04 07:52:53 on dist-test-slave-ghlz"
I20250904 07:52:53.944693 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:53.956760 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:53.957381 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:53.974854 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:43771
I20250904 07:52:53.974900 874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:43771 every 8 connection(s)
I20250904 07:52:53.976322 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:53.980082 875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4: Bootstrap starting.
I20250904 07:52:53.981985 875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:53.983639 875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4: No bootstrap required, opened a new log
I20250904 07:52:53.984283 875 raft_consensus.cc:357] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER }
I20250904 07:52:53.984411 875 raft_consensus.cc:383] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:53.984508 875 raft_consensus.cc:738] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dac51cffb94d80be7fb582eaeb48f4, State: Initialized, Role: FOLLOWER
I20250904 07:52:53.984727 875 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER }
I20250904 07:52:53.984898 875 raft_consensus.cc:397] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:53.984977 875 raft_consensus.cc:491] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:53.985095 875 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:53.986809 875 raft_consensus.cc:513] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER }
I20250904 07:52:53.987026 875 leader_election.cc:304] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dac51cffb94d80be7fb582eaeb48f4; no voters:
I20250904 07:52:53.987504 875 leader_election.cc:290] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:53.987643 878 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:53.988150 878 raft_consensus.cc:695] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 1 LEADER]: Becoming Leader. State: Replica: a0dac51cffb94d80be7fb582eaeb48f4, State: Running, Role: LEADER
I20250904 07:52:53.988513 878 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER }
I20250904 07:52:53.989676 875 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:53.996240 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER } }
I20250904 07:52:53.996429 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.997036 879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dac51cffb94d80be7fb582eaeb48f4. Latest consensus state: current_term: 1 leader_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dac51cffb94d80be7fb582eaeb48f4" member_type: VOTER } }
I20250904 07:52:53.997241 879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:53.997412 894 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:53.997715 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:53.999725 894 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:54.002629 894 catalog_manager.cc:1349] Generated new cluster ID: b08e3557681642b8932b36fcb79dbd21
I20250904 07:52:54.002723 894 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:54.027279 894 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:54.028371 894 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:54.040100 894 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4: Generated new TSK 0
I20250904 07:52:54.040339 894 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:54.062538 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:54.065646 897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.066219 898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.067515 900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.067729 413 server_base.cc:1047] running on GCE node
I20250904 07:52:54.068169 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:54.068279 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:54.068341 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972374068320 us; error 0 us; skew 500 ppm
I20250904 07:52:54.068539 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:54.069864 413 webserver.cc:480] Webserver started at http://127.0.103.65:44115/ using document root <none> and password file <none>
I20250904 07:52:54.070124 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:54.070233 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:54.070392 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:54.071183 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "4d3846af4d0a411f80bb1486fec46b3a"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.074906 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250904 07:52:54.076833 905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.077297 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:54.077446 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "4d3846af4d0a411f80bb1486fec46b3a"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.077555 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInvalidPredicates.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:54.101961 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:54.102604 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:54.104638 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:54.104744 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.104883 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:54.104960 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250904 07:52:54.153916 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35063
I20250904 07:52:54.153934 975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:35063 every 8 connection(s)
I20250904 07:52:54.155990 976 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43771
I20250904 07:52:54.156141 976 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:54.157042 976 heartbeater.cc:507] Master 127.0.103.126:43771 requested a full tablet report, sending...
I20250904 07:52:54.157946 830 ts_manager.cc:194] Registered new tserver with Master: 4d3846af4d0a411f80bb1486fec46b3a (127.0.103.65:35063)
I20250904 07:52:54.158506 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00340718s
I20250904 07:52:54.159653 830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52174
I20250904 07:52:54.171236 976 heartbeater.cc:499] Master 127.0.103.126:43771 was elected leader, sending a full tablet report...
I20250904 07:52:54.171566 830 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52190:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:54.188898 935 tablet_service.cc:1468] Processing CreateTablet for tablet 60a882d2a8ca47409e9fb03d67c4656e (DEFAULT_TABLE table=client-testtb [id=3a402aa0b5c04bca9db7030929f2f15b]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:54.188934 936 tablet_service.cc:1468] Processing CreateTablet for tablet d531a7d9d8124eccbb5cb9b2a6df1176 (DEFAULT_TABLE table=client-testtb [id=3a402aa0b5c04bca9db7030929f2f15b]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:54.189461 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a882d2a8ca47409e9fb03d67c4656e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.189744 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d531a7d9d8124eccbb5cb9b2a6df1176. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.197504 986 tablet_bootstrap.cc:492] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: Bootstrap starting.
I20250904 07:52:54.199729 986 tablet_bootstrap.cc:654] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.202387 986 tablet_bootstrap.cc:492] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: No bootstrap required, opened a new log
I20250904 07:52:54.202525 986 ts_tablet_manager.cc:1397] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250904 07:52:54.203339 986 raft_consensus.cc:357] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.203521 986 raft_consensus.cc:383] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.203620 986 raft_consensus.cc:738] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3846af4d0a411f80bb1486fec46b3a, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.203853 986 consensus_queue.cc:260] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.204068 986 raft_consensus.cc:397] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.204174 986 raft_consensus.cc:491] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.204306 986 raft_consensus.cc:3058] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.207269 986 raft_consensus.cc:513] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.207518 986 leader_election.cc:304] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3846af4d0a411f80bb1486fec46b3a; no voters:
I20250904 07:52:54.208010 986 leader_election.cc:290] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.208163 988 raft_consensus.cc:2802] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.209053 988 raft_consensus.cc:695] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 1 LEADER]: Becoming Leader. State: Replica: 4d3846af4d0a411f80bb1486fec46b3a, State: Running, Role: LEADER
I20250904 07:52:54.210068 988 consensus_queue.cc:237] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.212350 830 catalog_manager.cc:5582] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3846af4d0a411f80bb1486fec46b3a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4d3846af4d0a411f80bb1486fec46b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.216061 986 ts_tablet_manager.cc:1428] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20250904 07:52:54.216363 986 tablet_bootstrap.cc:492] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: Bootstrap starting.
I20250904 07:52:54.218189 986 tablet_bootstrap.cc:654] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.222572 986 tablet_bootstrap.cc:492] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: No bootstrap required, opened a new log
I20250904 07:52:54.222748 986 ts_tablet_manager.cc:1397] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250904 07:52:54.223410 986 raft_consensus.cc:357] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.223585 986 raft_consensus.cc:383] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.223702 986 raft_consensus.cc:738] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3846af4d0a411f80bb1486fec46b3a, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.224002 986 consensus_queue.cc:260] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.224151 986 raft_consensus.cc:397] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.224252 986 raft_consensus.cc:491] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.224364 986 raft_consensus.cc:3058] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.226663 986 raft_consensus.cc:513] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.226868 986 leader_election.cc:304] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3846af4d0a411f80bb1486fec46b3a; no voters:
I20250904 07:52:54.227092 986 leader_election.cc:290] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.227141 990 raft_consensus.cc:2802] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.227274 990 raft_consensus.cc:695] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 1 LEADER]: Becoming Leader. State: Replica: 4d3846af4d0a411f80bb1486fec46b3a, State: Running, Role: LEADER
I20250904 07:52:54.227702 986 ts_tablet_manager.cc:1428] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250904 07:52:54.227579 990 consensus_queue.cc:237] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } }
I20250904 07:52:54.229372 830 catalog_manager.cc:5582] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3846af4d0a411f80bb1486fec46b3a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4d3846af4d0a411f80bb1486fec46b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3846af4d0a411f80bb1486fec46b3a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35063 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.236687 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:54.247164 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:54.247568 413 tablet_replica.cc:331] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a: stopping tablet replica
I20250904 07:52:54.247758 413 raft_consensus.cc:2241] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.247938 413 raft_consensus.cc:2270] T d531a7d9d8124eccbb5cb9b2a6df1176 P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.248775 413 tablet_replica.cc:331] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a: stopping tablet replica
I20250904 07:52:54.248957 413 raft_consensus.cc:2241] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.249089 413 raft_consensus.cc:2270] T 60a882d2a8ca47409e9fb03d67c4656e P 4d3846af4d0a411f80bb1486fec46b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.263797 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:54.267737 413 master.cc:561] Master@127.0.103.126:43771 shutting down...
I20250904 07:52:54.278153 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.278337 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.278509 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P a0dac51cffb94d80be7fb582eaeb48f4: stopping tablet replica
I20250904 07:52:54.293444 413 master.cc:583] Master@127.0.103.126:43771 shutdown complete.
[ OK ] ClientTest.TestInvalidPredicates (370 ms)
[ RUN ] ClientTest.TestExclusiveInclusiveRangeBounds
I20250904 07:52:54.303129 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:33975
I20250904 07:52:54.303617 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:54.305903 995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.306110 996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.307170 413 server_base.cc:1047] running on GCE node
W20250904 07:52:54.307207 999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.307592 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:54.307700 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:54.307770 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972374307748 us; error 0 us; skew 500 ppm
I20250904 07:52:54.307967 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:54.309351 413 webserver.cc:480] Webserver started at http://127.0.103.126:34905/ using document root <none> and password file <none>
I20250904 07:52:54.309641 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:54.309736 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:54.309891 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:54.310669 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "732e97b2f160465ba0da4ed7a6f51bba"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.313928 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250904 07:52:54.315817 1003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.316193 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:54.316291 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "732e97b2f160465ba0da4ed7a6f51bba"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.316413 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:54.344038 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:54.344720 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:54.364240 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:33975
I20250904 07:52:54.364259 1064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:33975 every 8 connection(s)
I20250904 07:52:54.366421 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.372115 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: Bootstrap starting.
I20250904 07:52:54.373811 1065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.375790 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: No bootstrap required, opened a new log
I20250904 07:52:54.376567 1065 raft_consensus.cc:357] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER }
I20250904 07:52:54.376724 1065 raft_consensus.cc:383] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.376824 1065 raft_consensus.cc:738] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732e97b2f160465ba0da4ed7a6f51bba, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.377067 1065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER }
I20250904 07:52:54.377252 1065 raft_consensus.cc:397] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.377332 1065 raft_consensus.cc:491] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.377441 1065 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.380244 1065 raft_consensus.cc:513] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER }
I20250904 07:52:54.380468 1065 leader_election.cc:304] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 732e97b2f160465ba0da4ed7a6f51bba; no voters:
I20250904 07:52:54.380892 1065 leader_election.cc:290] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.381028 1068 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.381738 1068 raft_consensus.cc:695] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 1 LEADER]: Becoming Leader. State: Replica: 732e97b2f160465ba0da4ed7a6f51bba, State: Running, Role: LEADER
I20250904 07:52:54.381817 1065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:54.382021 1068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER }
I20250904 07:52:54.382879 1070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 732e97b2f160465ba0da4ed7a6f51bba. Latest consensus state: current_term: 1 leader_uuid: "732e97b2f160465ba0da4ed7a6f51bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER } }
I20250904 07:52:54.382854 1069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "732e97b2f160465ba0da4ed7a6f51bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732e97b2f160465ba0da4ed7a6f51bba" member_type: VOTER } }
I20250904 07:52:54.383075 1070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:54.383124 1069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:54.395646 1073 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:54.397380 1073 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:54.399960 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:54.401912 1073 catalog_manager.cc:1349] Generated new cluster ID: 9e81357240c544a78c5c9b1b7a59f24d
I20250904 07:52:54.402004 1073 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:54.430563 1073 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:54.431713 1073 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:54.437956 1073 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: Generated new TSK 0
I20250904 07:52:54.438172 1073 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:54.464951 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:54.467545 1086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.468480 1087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.468701 1089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.468804 413 server_base.cc:1047] running on GCE node
I20250904 07:52:54.469096 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:54.469162 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:54.469239 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972374469207 us; error 0 us; skew 500 ppm
I20250904 07:52:54.469900 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:54.470916 413 webserver.cc:480] Webserver started at http://127.0.103.65:39309/ using document root <none> and password file <none>
I20250904 07:52:54.471113 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:54.471205 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:54.471323 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:54.471911 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.474159 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:54.475543 1094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.475876 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:54.475968 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.476065 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:54.492733 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:54.493232 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:54.495100 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:54.495203 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.495312 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:54.495410 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.521099 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:46369
I20250904 07:52:54.521139 1164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:46369 every 8 connection(s)
I20250904 07:52:54.522562 1165 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33975
I20250904 07:52:54.522667 1165 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:54.523502 1165 heartbeater.cc:507] Master 127.0.103.126:33975 requested a full tablet report, sending...
I20250904 07:52:54.524199 1020 ts_manager.cc:194] Registered new tserver with Master: 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65:46369)
I20250904 07:52:54.524292 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002301292s
I20250904 07:52:54.525616 1020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55200
I20250904 07:52:54.534531 1020 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55222:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:54.536836 1165 heartbeater.cc:499] Master 127.0.103.126:33975 was elected leader, sending a full tablet report...
I20250904 07:52:54.547158 1124 tablet_service.cc:1468] Processing CreateTablet for tablet 026c8524c5df422783b22943592045d6 (DEFAULT_TABLE table=client-testtb [id=62039f3b3a6b46518ef6ced4cc777b4c]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:54.547135 1125 tablet_service.cc:1468] Processing CreateTablet for tablet de63566b6a3b4040b3173bce43c00ca3 (DEFAULT_TABLE table=client-testtb [id=62039f3b3a6b46518ef6ced4cc777b4c]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:54.547765 1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026c8524c5df422783b22943592045d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.547976 1125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de63566b6a3b4040b3173bce43c00ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.555101 1175 tablet_bootstrap.cc:492] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Bootstrap starting.
I20250904 07:52:54.556461 1175 tablet_bootstrap.cc:654] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.558243 1175 tablet_bootstrap.cc:492] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: No bootstrap required, opened a new log
I20250904 07:52:54.558346 1175 ts_tablet_manager.cc:1397] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20250904 07:52:54.558876 1175 raft_consensus.cc:357] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.559000 1175 raft_consensus.cc:383] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.559062 1175 raft_consensus.cc:738] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.559207 1175 consensus_queue.cc:260] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.559316 1175 raft_consensus.cc:397] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.559365 1175 raft_consensus.cc:491] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.559441 1175 raft_consensus.cc:3058] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.561460 1175 raft_consensus.cc:513] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.561676 1175 leader_election.cc:304] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb90e990bac4e9dbbe1cdcb2b8d71e6; no voters:
I20250904 07:52:54.562589 1175 leader_election.cc:290] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.562727 1177 raft_consensus.cc:2802] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.563275 1177 raft_consensus.cc:695] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Running, Role: LEADER
I20250904 07:52:54.563347 1175 ts_tablet_manager.cc:1428] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250904 07:52:54.563601 1175 tablet_bootstrap.cc:492] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Bootstrap starting.
I20250904 07:52:54.563602 1177 consensus_queue.cc:237] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.565342 1175 tablet_bootstrap.cc:654] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.565850 1019 catalog_manager.cc:5582] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.567209 1175 tablet_bootstrap.cc:492] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: No bootstrap required, opened a new log
I20250904 07:52:54.567380 1175 ts_tablet_manager.cc:1397] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:54.567996 1175 raft_consensus.cc:357] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.568156 1175 raft_consensus.cc:383] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.568243 1175 raft_consensus.cc:738] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.568552 1175 consensus_queue.cc:260] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.568742 1175 raft_consensus.cc:397] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.568893 1175 raft_consensus.cc:491] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.569015 1175 raft_consensus.cc:3058] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.571319 1175 raft_consensus.cc:513] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.571470 1175 leader_election.cc:304] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb90e990bac4e9dbbe1cdcb2b8d71e6; no voters:
I20250904 07:52:54.571624 1175 leader_election.cc:290] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.571681 1177 raft_consensus.cc:2802] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.571869 1177 raft_consensus.cc:695] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Running, Role: LEADER
I20250904 07:52:54.572448 1177 consensus_queue.cc:237] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.572587 1175 ts_tablet_manager.cc:1428] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250904 07:52:54.574349 1019 catalog_manager.cc:5582] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.586239 1019 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55222:
name: "TestExclusiveInclusiveRangeBounds"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\010\001\000\377\377\377\377\t\001\000c\000\000\000""\010\001\000\377\377\377\377\t\001\000c\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:54.592044 1124 tablet_service.cc:1468] Processing CreateTablet for tablet de013efc4ca540d992f3a1d01e2fc9da (DEFAULT_TABLE table=TestExclusiveInclusiveRangeBounds [id=61cb184308dc4e0b9cd56182f127136b]), partition=RANGE (key) PARTITION 0 <= VALUES < 100
I20250904 07:52:54.592434 1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de013efc4ca540d992f3a1d01e2fc9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.596832 1175 tablet_bootstrap.cc:492] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Bootstrap starting.
I20250904 07:52:54.598393 1175 tablet_bootstrap.cc:654] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.600543 1175 tablet_bootstrap.cc:492] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: No bootstrap required, opened a new log
I20250904 07:52:54.600687 1175 ts_tablet_manager.cc:1397] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:54.601500 1175 raft_consensus.cc:357] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.601644 1175 raft_consensus.cc:383] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.601752 1175 raft_consensus.cc:738] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.601948 1175 consensus_queue.cc:260] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.602092 1175 raft_consensus.cc:397] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.602165 1175 raft_consensus.cc:491] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.602270 1175 raft_consensus.cc:3058] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.603883 1175 raft_consensus.cc:513] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.604084 1175 leader_election.cc:304] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb90e990bac4e9dbbe1cdcb2b8d71e6; no voters:
I20250904 07:52:54.604240 1175 leader_election.cc:290] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.604358 1177 raft_consensus.cc:2802] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.604543 1177 raft_consensus.cc:695] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Running, Role: LEADER
I20250904 07:52:54.604655 1175 ts_tablet_manager.cc:1428] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:52:54.604785 1177 consensus_queue.cc:237] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.606881 1019 catalog_manager.cc:5582] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.611809 1019 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55222:
table { table_name: "TestExclusiveInclusiveRangeBounds" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "\010\001\000\307\000\000\000\t\001\000+\001\000\000""\010\001\000\307\000\000\000\t\001\000+\001\000\000" indirect_data: """" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: true
W20250904 07:52:54.615442 1019 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59614d85a4f643769958a436430a9a12 has no consensus state
I20250904 07:52:54.618538 1124 tablet_service.cc:1468] Processing CreateTablet for tablet 59614d85a4f643769958a436430a9a12 (DEFAULT_TABLE table=TestExclusiveInclusiveRangeBounds [id=61cb184308dc4e0b9cd56182f127136b]), partition=RANGE (key) PARTITION 200 <= VALUES < 300
I20250904 07:52:54.618917 1124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59614d85a4f643769958a436430a9a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.623282 1175 tablet_bootstrap.cc:492] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Bootstrap starting.
I20250904 07:52:54.624630 1175 tablet_bootstrap.cc:654] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.626772 1175 tablet_bootstrap.cc:492] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: No bootstrap required, opened a new log
I20250904 07:52:54.626909 1175 ts_tablet_manager.cc:1397] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:54.627429 1175 raft_consensus.cc:357] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.627578 1175 raft_consensus.cc:383] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.627653 1175 raft_consensus.cc:738] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.627888 1175 consensus_queue.cc:260] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.628027 1175 raft_consensus.cc:397] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.628126 1175 raft_consensus.cc:491] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.628228 1175 raft_consensus.cc:3058] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.629940 1175 raft_consensus.cc:513] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.630117 1175 leader_election.cc:304] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb90e990bac4e9dbbe1cdcb2b8d71e6; no voters:
I20250904 07:52:54.630275 1175 leader_election.cc:290] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.630347 1177 raft_consensus.cc:2802] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.630492 1177 raft_consensus.cc:695] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb90e990bac4e9dbbe1cdcb2b8d71e6, State: Running, Role: LEADER
I20250904 07:52:54.630703 1177 consensus_queue.cc:237] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } }
I20250904 07:52:54.630796 1175 ts_tablet_manager.cc:1428] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250904 07:52:54.632542 1020 catalog_manager.cc:5582] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb90e990bac4e9dbbe1cdcb2b8d71e6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46369 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.636261 1020 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: TestExclusiveInclusiveRangeBounds [id=61cb184308dc4e0b9cd56182f127136b] alter complete (version 0)
I20250904 07:52:54.683785 1020 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55222:
table { table_name: "TestExclusiveInclusiveRangeBounds" } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\006\001\000\000\000\000\000\007\001\000d\000\000\000""\006\001\000\000\000\000\000\007\001\000d\000\000\000" indirect_data: """" } } } alter_schema_steps { type: DROP_RANGE_PARTITION drop_range_partition { range_bounds { rows: "\010\001\000\307\000\000\000\t\001\000+\001\000\000""\010\001\000\307\000\000\000\t\001\000+\001\000\000" indirect_data: """" } } } schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: true
I20250904 07:52:54.687235 1020 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: Sending DeleteTablet for 1 replicas of tablet de013efc4ca540d992f3a1d01e2fc9da
I20250904 07:52:54.687610 1020 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: Sending DeleteTablet for 1 replicas of tablet 59614d85a4f643769958a436430a9a12
I20250904 07:52:54.688198 1125 tablet_service.cc:1515] Processing DeleteTablet for tablet 59614d85a4f643769958a436430a9a12 with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-09-04 07:52:54 UTC) from {username='slave'} at 127.0.0.1:59922
I20250904 07:52:54.688117 1124 tablet_service.cc:1515] Processing DeleteTablet for tablet de013efc4ca540d992f3a1d01e2fc9da with delete_type TABLET_DATA_DELETED (Partition dropped at 2025-09-04 07:52:54 UTC) from {username='slave'} at 127.0.0.1:59922
I20250904 07:52:54.689148 1190 tablet_replica.cc:331] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: stopping tablet replica
I20250904 07:52:54.689337 1190 raft_consensus.cc:2241] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.690057 1190 raft_consensus.cc:2270] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.690979 1190 ts_tablet_manager.cc:1905] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 07:52:54.691920 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:54.699229 1190 ts_tablet_manager.cc:1918] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250904 07:52:54.699522 1190 log.cc:1199] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting WAL directory at /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root/wals/59614d85a4f643769958a436430a9a12
I20250904 07:52:54.700275 1190 ts_tablet_manager.cc:1939] T 59614d85a4f643769958a436430a9a12 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting consensus metadata
I20250904 07:52:54.701926 1190 tablet_replica.cc:331] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: stopping tablet replica
I20250904 07:52:54.701995 1006 catalog_manager.cc:4928] TS 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65:46369): tablet 59614d85a4f643769958a436430a9a12 (table TestExclusiveInclusiveRangeBounds [id=61cb184308dc4e0b9cd56182f127136b]) successfully deleted
I20250904 07:52:54.702203 1190 raft_consensus.cc:2241] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.702308 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:54.702396 1190 raft_consensus.cc:2270] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.703150 1190 ts_tablet_manager.cc:1905] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 07:52:54.706903 1190 ts_tablet_manager.cc:1918] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250904 07:52:54.707028 1190 log.cc:1199] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting WAL directory at /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestExclusiveInclusiveRangeBounds.1756972373064810-413-0/minicluster-data/ts-0-root/wals/de013efc4ca540d992f3a1d01e2fc9da
I20250904 07:52:54.707345 1190 ts_tablet_manager.cc:1939] T de013efc4ca540d992f3a1d01e2fc9da P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: Deleting consensus metadata
I20250904 07:52:54.708101 413 tablet_replica.cc:331] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: stopping tablet replica
W20250904 07:52:54.708061 1006 catalog_manager.cc:4920] TS 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65:46369): delete failed for tablet de013efc4ca540d992f3a1d01e2fc9da with error code TABLET_NOT_RUNNING: Service unavailable: Tablet Manager is not running: MANAGER_QUIESCING
I20250904 07:52:54.708281 413 raft_consensus.cc:2241] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.708425 413 raft_consensus.cc:2270] T 026c8524c5df422783b22943592045d6 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.709172 413 tablet_replica.cc:331] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6: stopping tablet replica
I20250904 07:52:54.709293 413 raft_consensus.cc:2241] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.709403 413 raft_consensus.cc:2270] T de63566b6a3b4040b3173bce43c00ca3 P 1cb90e990bac4e9dbbe1cdcb2b8d71e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.723923 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:54.728097 413 master.cc:561] Master@127.0.103.126:33975 shutting down...
W20250904 07:52:54.747807 1006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46369: connect: Connection refused (error 111)
W20250904 07:52:54.748919 1006 catalog_manager.cc:4671] TS 1cb90e990bac4e9dbbe1cdcb2b8d71e6 (127.0.103.65:46369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de013efc4ca540d992f3a1d01e2fc9da: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46369: connect: Connection refused (error 111)
I20250904 07:52:54.755249 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.755473 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.755681 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 732e97b2f160465ba0da4ed7a6f51bba: stopping tablet replica
I20250904 07:52:54.759835 413 master.cc:583] Master@127.0.103.126:33975 shutdown complete.
[ OK ] ClientTest.TestExclusiveInclusiveRangeBounds (465 ms)
[ RUN ] ClientTest.TestBasicIdBasedLookup
I20250904 07:52:54.768151 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46841
I20250904 07:52:54.768553 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:54.770577 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.770912 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.771250 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.771337 413 server_base.cc:1047] running on GCE node
I20250904 07:52:54.771636 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:54.771695 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:54.771740 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972374771727 us; error 0 us; skew 500 ppm
I20250904 07:52:54.771868 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:54.772826 413 webserver.cc:480] Webserver started at http://127.0.103.126:36189/ using document root <none> and password file <none>
I20250904 07:52:54.773082 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:54.773146 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:54.773248 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:54.773761 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "33b560a27b9b4267afedc28a0a27f3ef"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.775839 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:54.777215 1200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.777474 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:54.777580 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "33b560a27b9b4267afedc28a0a27f3ef"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.777666 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:54.797444 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:54.797943 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:54.811630 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46841
I20250904 07:52:54.811651 1261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46841 every 8 connection(s)
I20250904 07:52:54.812707 1262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.816437 1262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef: Bootstrap starting.
I20250904 07:52:54.817703 1262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.819090 1262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef: No bootstrap required, opened a new log
I20250904 07:52:54.819667 1262 raft_consensus.cc:357] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER }
I20250904 07:52:54.819800 1262 raft_consensus.cc:383] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.819892 1262 raft_consensus.cc:738] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b560a27b9b4267afedc28a0a27f3ef, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.820093 1262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER }
I20250904 07:52:54.820223 1262 raft_consensus.cc:397] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.820292 1262 raft_consensus.cc:491] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.820397 1262 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.822055 1262 raft_consensus.cc:513] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER }
I20250904 07:52:54.822257 1262 leader_election.cc:304] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b560a27b9b4267afedc28a0a27f3ef; no voters:
I20250904 07:52:54.822600 1262 leader_election.cc:290] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.822731 1265 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.823210 1265 raft_consensus.cc:695] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 1 LEADER]: Becoming Leader. State: Replica: 33b560a27b9b4267afedc28a0a27f3ef, State: Running, Role: LEADER
I20250904 07:52:54.823486 1262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:54.823505 1265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER }
I20250904 07:52:54.824441 1267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b560a27b9b4267afedc28a0a27f3ef. Latest consensus state: current_term: 1 leader_uuid: "33b560a27b9b4267afedc28a0a27f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER } }
I20250904 07:52:54.824396 1266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b560a27b9b4267afedc28a0a27f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b560a27b9b4267afedc28a0a27f3ef" member_type: VOTER } }
I20250904 07:52:54.824681 1266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:54.824945 1267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:54.825837 1270 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:54.827090 1270 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:54.829975 1270 catalog_manager.cc:1349] Generated new cluster ID: 31494832052e4c3fbabd686708bc8f5e
I20250904 07:52:54.830068 1270 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:54.831280 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:54.837461 1270 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:54.838142 1270 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:54.847173 1270 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef: Generated new TSK 0
I20250904 07:52:54.847365 1270 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:54.863664 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:54.866274 1284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.866657 1283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:54.866763 1286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:54.866909 413 server_base.cc:1047] running on GCE node
I20250904 07:52:54.867235 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:54.867300 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:54.867369 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972374867337 us; error 0 us; skew 500 ppm
I20250904 07:52:54.867507 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:54.868515 413 webserver.cc:480] Webserver started at http://127.0.103.65:41039/ using document root <none> and password file <none>
I20250904 07:52:54.868708 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:54.868798 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:54.868966 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:54.869501 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "55a3bc3472bf4cd98828f9c301d24230"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.871570 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:52:54.873116 1291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.873483 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:54.873601 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "55a3bc3472bf4cd98828f9c301d24230"
format_stamp: "Formatted at 2025-09-04 07:52:54 on dist-test-slave-ghlz"
I20250904 07:52:54.873725 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBasicIdBasedLookup.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:54.891357 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:54.891849 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:54.893343 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:54.893446 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.893584 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:54.893671 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:54.918649 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39779
I20250904 07:52:54.918705 1361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39779 every 8 connection(s)
I20250904 07:52:54.920522 1362 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46841
I20250904 07:52:54.920632 1362 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:54.921023 1362 heartbeater.cc:507] Master 127.0.103.126:46841 requested a full tablet report, sending...
I20250904 07:52:54.921679 1217 ts_manager.cc:194] Registered new tserver with Master: 55a3bc3472bf4cd98828f9c301d24230 (127.0.103.65:39779)
I20250904 07:52:54.922214 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002288185s
I20250904 07:52:54.922663 1217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39686
I20250904 07:52:54.931891 1217 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39702:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:54.933780 1362 heartbeater.cc:499] Master 127.0.103.126:46841 was elected leader, sending a full tablet report...
I20250904 07:52:54.944550 1326 tablet_service.cc:1468] Processing CreateTablet for tablet 6fe7c52e127543739fb6ec54dddc238a (DEFAULT_TABLE table=client-testtb [id=2a5c6af22b4c4c5e88b826b0e5e97666]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:54.944645 1324 tablet_service.cc:1468] Processing CreateTablet for tablet 22f86160ee3e419aaed68bca46ebf09c (DEFAULT_TABLE table=client-testtb [id=2a5c6af22b4c4c5e88b826b0e5e97666]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:54.945020 1326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe7c52e127543739fb6ec54dddc238a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.945269 1324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f86160ee3e419aaed68bca46ebf09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:54.951915 1372 tablet_bootstrap.cc:492] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: Bootstrap starting.
I20250904 07:52:54.953222 1372 tablet_bootstrap.cc:654] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.954733 1372 tablet_bootstrap.cc:492] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: No bootstrap required, opened a new log
I20250904 07:52:54.954844 1372 ts_tablet_manager.cc:1397] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:52:54.955406 1372 raft_consensus.cc:357] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.955528 1372 raft_consensus.cc:383] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.955590 1372 raft_consensus.cc:738] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a3bc3472bf4cd98828f9c301d24230, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.955780 1372 consensus_queue.cc:260] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.955914 1372 raft_consensus.cc:397] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.956012 1372 raft_consensus.cc:491] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.956142 1372 raft_consensus.cc:3058] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.958295 1372 raft_consensus.cc:513] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.958467 1372 leader_election.cc:304] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a3bc3472bf4cd98828f9c301d24230; no voters:
I20250904 07:52:54.958796 1372 leader_election.cc:290] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.958925 1374 raft_consensus.cc:2802] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.959444 1374 raft_consensus.cc:695] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 1 LEADER]: Becoming Leader. State: Replica: 55a3bc3472bf4cd98828f9c301d24230, State: Running, Role: LEADER
I20250904 07:52:54.959533 1372 ts_tablet_manager.cc:1428] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250904 07:52:54.959749 1372 tablet_bootstrap.cc:492] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: Bootstrap starting.
I20250904 07:52:54.959767 1374 consensus_queue.cc:237] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.961535 1372 tablet_bootstrap.cc:654] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:54.962297 1216 catalog_manager.cc:5582] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a3bc3472bf4cd98828f9c301d24230 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "55a3bc3472bf4cd98828f9c301d24230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:54.963451 1372 tablet_bootstrap.cc:492] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: No bootstrap required, opened a new log
I20250904 07:52:54.963672 1372 ts_tablet_manager.cc:1397] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:54.964488 1372 raft_consensus.cc:357] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.964672 1372 raft_consensus.cc:383] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:54.964749 1372 raft_consensus.cc:738] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a3bc3472bf4cd98828f9c301d24230, State: Initialized, Role: FOLLOWER
I20250904 07:52:54.965036 1372 consensus_queue.cc:260] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.965210 1372 raft_consensus.cc:397] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:54.965303 1372 raft_consensus.cc:491] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:54.965399 1372 raft_consensus.cc:3058] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:54.967648 1372 raft_consensus.cc:513] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.967826 1372 leader_election.cc:304] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a3bc3472bf4cd98828f9c301d24230; no voters:
I20250904 07:52:54.968003 1372 leader_election.cc:290] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:54.968044 1374 raft_consensus.cc:2802] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:54.968222 1374 raft_consensus.cc:695] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 1 LEADER]: Becoming Leader. State: Replica: 55a3bc3472bf4cd98828f9c301d24230, State: Running, Role: LEADER
I20250904 07:52:54.968386 1372 ts_tablet_manager.cc:1428] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250904 07:52:54.968602 1374 consensus_queue.cc:237] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } }
I20250904 07:52:54.970212 1216 catalog_manager.cc:5582] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a3bc3472bf4cd98828f9c301d24230 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "55a3bc3472bf4cd98828f9c301d24230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a3bc3472bf4cd98828f9c301d24230" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39779 } health_report { overall_health: HEALTHY } } }
W20250904 07:52:54.983534 1367 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'dummy-tablet-id', attempt: 1 } failed
I20250904 07:52:54.984238 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:54.991619 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:54.991937 413 tablet_replica.cc:331] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230: stopping tablet replica
I20250904 07:52:54.992133 413 raft_consensus.cc:2241] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.992261 413 raft_consensus.cc:2270] T 6fe7c52e127543739fb6ec54dddc238a P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:54.992959 413 tablet_replica.cc:331] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230: stopping tablet replica
I20250904 07:52:54.993062 413 raft_consensus.cc:2241] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:54.993185 413 raft_consensus.cc:2270] T 22f86160ee3e419aaed68bca46ebf09c P 55a3bc3472bf4cd98828f9c301d24230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:55.007035 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:55.010782 413 master.cc:561] Master@127.0.103.126:46841 shutting down...
I20250904 07:52:55.018556 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:55.018729 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:55.018868 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 33b560a27b9b4267afedc28a0a27f3ef: stopping tablet replica
I20250904 07:52:55.032473 413 master.cc:583] Master@127.0.103.126:46841 shutdown complete.
[ OK ] ClientTest.TestBasicIdBasedLookup (273 ms)
[ RUN ] ClientTest.TestScannerKeepAlive
I20250904 07:52:55.041512 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37871
I20250904 07:52:55.041978 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:55.043990 1381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:55.044023 1382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:55.044483 1384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:55.045365 413 server_base.cc:1047] running on GCE node
I20250904 07:52:55.045599 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:55.045660 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:55.045742 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972375045710 us; error 0 us; skew 500 ppm
I20250904 07:52:55.045903 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:55.046963 413 webserver.cc:480] Webserver started at http://127.0.103.126:44047/ using document root <none> and password file <none>
I20250904 07:52:55.047154 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:55.047257 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:55.047375 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:55.047911 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "d00c9cfb821c40d2813bf5b93328e4f6"
format_stamp: "Formatted at 2025-09-04 07:52:55 on dist-test-slave-ghlz"
I20250904 07:52:55.050077 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:55.051469 1389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:55.051807 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:55.051918 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d00c9cfb821c40d2813bf5b93328e4f6"
format_stamp: "Formatted at 2025-09-04 07:52:55 on dist-test-slave-ghlz"
I20250904 07:52:55.052037 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:55.058795 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:55.059276 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:55.074162 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37871
I20250904 07:52:55.074210 1450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:37871 every 8 connection(s)
I20250904 07:52:55.075281 1451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:55.078850 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6: Bootstrap starting.
I20250904 07:52:55.080102 1451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:55.081537 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6: No bootstrap required, opened a new log
I20250904 07:52:55.082173 1451 raft_consensus.cc:357] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER }
I20250904 07:52:55.082293 1451 raft_consensus.cc:383] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:55.082386 1451 raft_consensus.cc:738] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00c9cfb821c40d2813bf5b93328e4f6, State: Initialized, Role: FOLLOWER
I20250904 07:52:55.082572 1451 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER }
I20250904 07:52:55.082705 1451 raft_consensus.cc:397] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:55.082787 1451 raft_consensus.cc:491] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:55.083055 1451 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:55.084686 1451 raft_consensus.cc:513] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER }
I20250904 07:52:55.084905 1451 leader_election.cc:304] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00c9cfb821c40d2813bf5b93328e4f6; no voters:
I20250904 07:52:55.085291 1451 leader_election.cc:290] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:55.085383 1454 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:55.085888 1454 raft_consensus.cc:695] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 1 LEADER]: Becoming Leader. State: Replica: d00c9cfb821c40d2813bf5b93328e4f6, State: Running, Role: LEADER
I20250904 07:52:55.086150 1451 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:55.086169 1454 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER }
I20250904 07:52:55.087091 1456 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00c9cfb821c40d2813bf5b93328e4f6. Latest consensus state: current_term: 1 leader_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER } }
I20250904 07:52:55.087147 1455 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00c9cfb821c40d2813bf5b93328e4f6" member_type: VOTER } }
I20250904 07:52:55.087337 1456 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:55.087364 1455 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:55.088591 1459 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:55.090217 1459 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:55.091516 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:55.093041 1459 catalog_manager.cc:1349] Generated new cluster ID: 3f0ac07861304efe9295c4ec97bf8e4f
I20250904 07:52:55.093125 1459 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:55.101462 1459 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:55.102137 1459 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:55.112803 1459 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6: Generated new TSK 0
I20250904 07:52:55.113030 1459 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:55.123939 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:55.126499 1472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:55.126521 1473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:55.128214 413 server_base.cc:1047] running on GCE node
W20250904 07:52:55.128238 1475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:55.128602 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:55.128696 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:55.128755 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972375128738 us; error 0 us; skew 500 ppm
I20250904 07:52:55.128953 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:55.130091 413 webserver.cc:480] Webserver started at http://127.0.103.65:38991/ using document root <none> and password file <none>
I20250904 07:52:55.130304 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:55.130380 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:55.130520 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:55.131070 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "e1599ed4f48443cfa2c97f599c8e046b"
format_stamp: "Formatted at 2025-09-04 07:52:55 on dist-test-slave-ghlz"
I20250904 07:52:55.133278 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:52:55.134658 1480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:55.134987 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:55.135106 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "e1599ed4f48443cfa2c97f599c8e046b"
format_stamp: "Formatted at 2025-09-04 07:52:55 on dist-test-slave-ghlz"
I20250904 07:52:55.135203 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestScannerKeepAlive.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:55.149195 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:55.149695 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:55.151187 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:55.151281 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:55.151512 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:55.152920 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250904 07:52:55.177811 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41607
I20250904 07:52:55.177847 1550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41607 every 8 connection(s)
I20250904 07:52:55.179260 1551 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37871
I20250904 07:52:55.179374 1551 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:55.179756 1551 heartbeater.cc:507] Master 127.0.103.126:37871 requested a full tablet report, sending...
I20250904 07:52:55.180488 1406 ts_manager.cc:194] Registered new tserver with Master: e1599ed4f48443cfa2c97f599c8e046b (127.0.103.65:41607)
I20250904 07:52:55.180939 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002270763s
I20250904 07:52:55.181567 1406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40666
I20250904 07:52:55.191404 1406 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40694:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:55.192786 1551 heartbeater.cc:499] Master 127.0.103.126:37871 was elected leader, sending a full tablet report...
I20250904 07:52:55.204829 1511 tablet_service.cc:1468] Processing CreateTablet for tablet e261fd03589b4fd29db41cf067af7689 (DEFAULT_TABLE table=client-testtb [id=f833147243494f05bea11332bfd4e902]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:55.204895 1510 tablet_service.cc:1468] Processing CreateTablet for tablet cbb01dc6a9284bc9b4cf67690404fb16 (DEFAULT_TABLE table=client-testtb [id=f833147243494f05bea11332bfd4e902]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:55.205435 1510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbb01dc6a9284bc9b4cf67690404fb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:55.205713 1511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e261fd03589b4fd29db41cf067af7689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:55.210597 1561 tablet_bootstrap.cc:492] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: Bootstrap starting.
I20250904 07:52:55.212106 1561 tablet_bootstrap.cc:654] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:55.214120 1561 tablet_bootstrap.cc:492] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: No bootstrap required, opened a new log
I20250904 07:52:55.214224 1561 ts_tablet_manager.cc:1397] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:55.214748 1561 raft_consensus.cc:357] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.214879 1561 raft_consensus.cc:383] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:55.214949 1561 raft_consensus.cc:738] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1599ed4f48443cfa2c97f599c8e046b, State: Initialized, Role: FOLLOWER
I20250904 07:52:55.215098 1561 consensus_queue.cc:260] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.215217 1561 raft_consensus.cc:397] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:55.215270 1561 raft_consensus.cc:491] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:55.215348 1561 raft_consensus.cc:3058] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:55.217428 1561 raft_consensus.cc:513] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.217607 1561 leader_election.cc:304] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1599ed4f48443cfa2c97f599c8e046b; no voters:
I20250904 07:52:55.217980 1561 leader_election.cc:290] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:55.218132 1563 raft_consensus.cc:2802] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:55.218849 1561 ts_tablet_manager.cc:1428] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250904 07:52:55.218888 1563 raft_consensus.cc:695] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 1 LEADER]: Becoming Leader. State: Replica: e1599ed4f48443cfa2c97f599c8e046b, State: Running, Role: LEADER
I20250904 07:52:55.219154 1561 tablet_bootstrap.cc:492] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: Bootstrap starting.
I20250904 07:52:55.219252 1563 consensus_queue.cc:237] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.220810 1561 tablet_bootstrap.cc:654] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:55.221482 1406 catalog_manager.cc:5582] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b reported cstate change: term changed from 0 to 1, leader changed from <none> to e1599ed4f48443cfa2c97f599c8e046b (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e1599ed4f48443cfa2c97f599c8e046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:55.222950 1561 tablet_bootstrap.cc:492] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: No bootstrap required, opened a new log
I20250904 07:52:55.223105 1561 ts_tablet_manager.cc:1397] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:55.223791 1561 raft_consensus.cc:357] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.223943 1561 raft_consensus.cc:383] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:55.224021 1561 raft_consensus.cc:738] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1599ed4f48443cfa2c97f599c8e046b, State: Initialized, Role: FOLLOWER
I20250904 07:52:55.224265 1561 consensus_queue.cc:260] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.224387 1561 raft_consensus.cc:397] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:55.224498 1561 raft_consensus.cc:491] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:55.224607 1561 raft_consensus.cc:3058] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:55.226950 1561 raft_consensus.cc:513] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.227114 1561 leader_election.cc:304] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1599ed4f48443cfa2c97f599c8e046b; no voters:
I20250904 07:52:55.227303 1561 leader_election.cc:290] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:55.227357 1563 raft_consensus.cc:2802] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:55.227532 1563 raft_consensus.cc:695] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 1 LEADER]: Becoming Leader. State: Replica: e1599ed4f48443cfa2c97f599c8e046b, State: Running, Role: LEADER
I20250904 07:52:55.227722 1561 ts_tablet_manager.cc:1428] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250904 07:52:55.227746 1563 consensus_queue.cc:237] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } }
I20250904 07:52:55.229496 1406 catalog_manager.cc:5582] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b reported cstate change: term changed from 0 to 1, leader changed from <none> to e1599ed4f48443cfa2c97f599c8e046b (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e1599ed4f48443cfa2c97f599c8e046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1599ed4f48443cfa2c97f599c8e046b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41607 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:56.866969 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20250904 07:52:56.867409 1556 client.cc:2040] Couldn't close scanner beddbba9e6a04ad1b5eb9793b0cce7ec: Service unavailable: reactor is shutting down (error 108)
W20250904 07:52:56.868520 1483 connection.cc:537] server connection from 127.0.0.1:34170 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250904 07:52:56.875697 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:56.876019 413 tablet_replica.cc:331] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b: stopping tablet replica
I20250904 07:52:56.876194 413 raft_consensus.cc:2241] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:56.876428 413 raft_consensus.cc:2270] T e261fd03589b4fd29db41cf067af7689 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:56.877029 413 tablet_replica.cc:331] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b: stopping tablet replica
I20250904 07:52:56.877188 413 raft_consensus.cc:2241] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:56.877354 413 raft_consensus.cc:2270] T cbb01dc6a9284bc9b4cf67690404fb16 P e1599ed4f48443cfa2c97f599c8e046b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:56.890525 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:56.894248 413 master.cc:561] Master@127.0.103.126:37871 shutting down...
I20250904 07:52:56.902058 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:56.902242 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:56.902372 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d00c9cfb821c40d2813bf5b93328e4f6: stopping tablet replica
I20250904 07:52:56.915078 413 master.cc:583] Master@127.0.103.126:37871 shutdown complete.
[ OK ] ClientTest.TestScannerKeepAlive (1882 ms)
[ RUN ] ClientTest.TestInsertAutoFlushSync
I20250904 07:52:56.923775 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:43463
I20250904 07:52:56.924265 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:56.926298 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:56.926775 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:56.926946 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:56.927466 413 server_base.cc:1047] running on GCE node
I20250904 07:52:56.927700 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:56.927779 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:56.927835 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972376927818 us; error 0 us; skew 500 ppm
I20250904 07:52:56.928011 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:56.929102 413 webserver.cc:480] Webserver started at http://127.0.103.126:39591/ using document root <none> and password file <none>
I20250904 07:52:56.929309 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:56.929386 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:56.929530 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:56.930125 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "d8909e39c0264214b9e8cbe487dbe7e2"
format_stamp: "Formatted at 2025-09-04 07:52:56 on dist-test-slave-ghlz"
I20250904 07:52:56.932219 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250904 07:52:56.933652 1581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:56.933969 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:56.934089 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d8909e39c0264214b9e8cbe487dbe7e2"
format_stamp: "Formatted at 2025-09-04 07:52:56 on dist-test-slave-ghlz"
I20250904 07:52:56.934190 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:56.951164 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:56.951678 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:56.964654 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:43463
I20250904 07:52:56.964660 1642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:43463 every 8 connection(s)
I20250904 07:52:56.965871 1643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:56.969499 1643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2: Bootstrap starting.
I20250904 07:52:56.970741 1643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:56.972141 1643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2: No bootstrap required, opened a new log
I20250904 07:52:56.972716 1643 raft_consensus.cc:357] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER }
I20250904 07:52:56.972841 1643 raft_consensus.cc:383] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:56.972958 1643 raft_consensus.cc:738] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8909e39c0264214b9e8cbe487dbe7e2, State: Initialized, Role: FOLLOWER
I20250904 07:52:56.973166 1643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER }
I20250904 07:52:56.973315 1643 raft_consensus.cc:397] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:56.973389 1643 raft_consensus.cc:491] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:56.973502 1643 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:56.975016 1643 raft_consensus.cc:513] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER }
I20250904 07:52:56.975207 1643 leader_election.cc:304] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8909e39c0264214b9e8cbe487dbe7e2; no voters:
I20250904 07:52:56.975538 1643 leader_election.cc:290] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:56.975685 1646 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:56.976161 1646 raft_consensus.cc:695] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 1 LEADER]: Becoming Leader. State: Replica: d8909e39c0264214b9e8cbe487dbe7e2, State: Running, Role: LEADER
I20250904 07:52:56.976369 1643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:56.976416 1646 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER }
I20250904 07:52:56.977301 1647 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER } }
I20250904 07:52:56.977370 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8909e39c0264214b9e8cbe487dbe7e2. Latest consensus state: current_term: 1 leader_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8909e39c0264214b9e8cbe487dbe7e2" member_type: VOTER } }
I20250904 07:52:56.977604 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:56.978510 1647 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:56.978991 1652 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:56.980428 1652 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:56.981127 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:56.983129 1652 catalog_manager.cc:1349] Generated new cluster ID: 4560b3712a9a477ab51d0cd5246a2527
I20250904 07:52:56.983209 1652 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:57.006737 1652 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:57.007439 1652 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:57.013960 1652 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2: Generated new TSK 0
I20250904 07:52:57.014154 1652 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:57.046032 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:57.049140 1664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.049467 1665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.049830 1667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:57.050455 413 server_base.cc:1047] running on GCE node
I20250904 07:52:57.050724 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:57.050786 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:57.050849 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972377050816 us; error 0 us; skew 500 ppm
I20250904 07:52:57.050999 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:57.052075 413 webserver.cc:480] Webserver started at http://127.0.103.65:32803/ using document root <none> and password file <none>
I20250904 07:52:57.052276 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:57.052369 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:57.052495 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:57.053162 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "4ecaebcb7d014fe8b50331d1b192bd06"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.055562 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:52:57.057088 1672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.057502 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:57.057592 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "4ecaebcb7d014fe8b50331d1b192bd06"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.057694 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertAutoFlushSync.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:57.087307 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:57.087844 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:57.089921 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:57.090098 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.090214 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:57.090301 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.113476 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40297
I20250904 07:52:57.113547 1742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40297 every 8 connection(s)
I20250904 07:52:57.115303 1743 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43463
I20250904 07:52:57.115453 1743 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:57.115983 1743 heartbeater.cc:507] Master 127.0.103.126:43463 requested a full tablet report, sending...
I20250904 07:52:57.117367 1598 ts_manager.cc:194] Registered new tserver with Master: 4ecaebcb7d014fe8b50331d1b192bd06 (127.0.103.65:40297)
I20250904 07:52:57.118119 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00348099s
I20250904 07:52:57.118592 1598 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51604
I20250904 07:52:57.128237 1598 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51620:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:57.129881 1743 heartbeater.cc:499] Master 127.0.103.126:43463 was elected leader, sending a full tablet report...
I20250904 07:52:57.143019 1703 tablet_service.cc:1468] Processing CreateTablet for tablet 4aa9e30fe890491db5034a2c7dda35ce (DEFAULT_TABLE table=client-testtb [id=6d07425a7ec54a62a596e5551b334e35]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:57.143085 1702 tablet_service.cc:1468] Processing CreateTablet for tablet 98c8b3900c3840eaa1fb29c007a81065 (DEFAULT_TABLE table=client-testtb [id=6d07425a7ec54a62a596e5551b334e35]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:57.143460 1703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa9e30fe890491db5034a2c7dda35ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.143709 1702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c8b3900c3840eaa1fb29c007a81065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.148522 1753 tablet_bootstrap.cc:492] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: Bootstrap starting.
I20250904 07:52:57.150379 1753 tablet_bootstrap.cc:654] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.151898 1753 tablet_bootstrap.cc:492] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: No bootstrap required, opened a new log
I20250904 07:52:57.152026 1753 ts_tablet_manager.cc:1397] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:57.152690 1753 raft_consensus.cc:357] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.152943 1753 raft_consensus.cc:383] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.153059 1753 raft_consensus.cc:738] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecaebcb7d014fe8b50331d1b192bd06, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.153368 1753 consensus_queue.cc:260] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.153592 1753 raft_consensus.cc:397] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.153695 1753 raft_consensus.cc:491] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.153844 1753 raft_consensus.cc:3058] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.155511 1753 raft_consensus.cc:513] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.155716 1753 leader_election.cc:304] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecaebcb7d014fe8b50331d1b192bd06; no voters:
I20250904 07:52:57.156123 1753 leader_election.cc:290] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.156323 1755 raft_consensus.cc:2802] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.156812 1753 ts_tablet_manager.cc:1428] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20250904 07:52:57.157029 1755 raft_consensus.cc:695] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecaebcb7d014fe8b50331d1b192bd06, State: Running, Role: LEADER
I20250904 07:52:57.157137 1753 tablet_bootstrap.cc:492] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: Bootstrap starting.
I20250904 07:52:57.157302 1755 consensus_queue.cc:237] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.159346 1753 tablet_bootstrap.cc:654] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.160511 1598 catalog_manager.cc:5582] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecaebcb7d014fe8b50331d1b192bd06 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:57.161509 1753 tablet_bootstrap.cc:492] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: No bootstrap required, opened a new log
I20250904 07:52:57.161638 1753 ts_tablet_manager.cc:1397] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:52:57.162209 1753 raft_consensus.cc:357] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.162380 1753 raft_consensus.cc:383] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.162472 1753 raft_consensus.cc:738] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecaebcb7d014fe8b50331d1b192bd06, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.162715 1753 consensus_queue.cc:260] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.162850 1753 raft_consensus.cc:397] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.162953 1753 raft_consensus.cc:491] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.163051 1753 raft_consensus.cc:3058] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.165449 1753 raft_consensus.cc:513] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.165629 1753 leader_election.cc:304] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecaebcb7d014fe8b50331d1b192bd06; no voters:
I20250904 07:52:57.165828 1753 leader_election.cc:290] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.165875 1755 raft_consensus.cc:2802] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.166081 1755 raft_consensus.cc:695] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecaebcb7d014fe8b50331d1b192bd06, State: Running, Role: LEADER
I20250904 07:52:57.166298 1755 consensus_queue.cc:237] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } }
I20250904 07:52:57.166296 1753 ts_tablet_manager.cc:1428] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250904 07:52:57.168035 1598 catalog_manager.cc:5582] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecaebcb7d014fe8b50331d1b192bd06 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaebcb7d014fe8b50331d1b192bd06" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40297 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:57.427654 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:57.435475 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:57.435837 413 tablet_replica.cc:331] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06: stopping tablet replica
I20250904 07:52:57.436034 413 raft_consensus.cc:2241] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.436254 413 raft_consensus.cc:2270] T 98c8b3900c3840eaa1fb29c007a81065 P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.437040 413 tablet_replica.cc:331] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06: stopping tablet replica
I20250904 07:52:57.437173 413 raft_consensus.cc:2241] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.437338 413 raft_consensus.cc:2270] T 4aa9e30fe890491db5034a2c7dda35ce P 4ecaebcb7d014fe8b50331d1b192bd06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.451231 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:57.454787 413 master.cc:561] Master@127.0.103.126:43463 shutting down...
I20250904 07:52:57.462457 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.462621 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.462795 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d8909e39c0264214b9e8cbe487dbe7e2: stopping tablet replica
I20250904 07:52:57.476217 413 master.cc:583] Master@127.0.103.126:43463 shutdown complete.
[ OK ] ClientTest.TestInsertAutoFlushSync (560 ms)
[ RUN ] ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn
I20250904 07:52:57.484444 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37245
I20250904 07:52:57.484850 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:57.486832 1764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.487083 1765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.487584 1767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:57.488039 413 server_base.cc:1047] running on GCE node
I20250904 07:52:57.488256 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:57.488327 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:57.488384 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972377488366 us; error 0 us; skew 500 ppm
I20250904 07:52:57.488562 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:57.489615 413 webserver.cc:480] Webserver started at http://127.0.103.126:45755/ using document root <none> and password file <none>
I20250904 07:52:57.489823 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:57.489890 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:57.490031 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:57.490572 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "cf46dde08ed84084819ee97e5b15916b"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.492587 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:57.494068 1772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.494374 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:52:57.494472 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "cf46dde08ed84084819ee97e5b15916b"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.494566 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:57.525218 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:57.525696 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:57.538291 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37245
I20250904 07:52:57.538313 1833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:37245 every 8 connection(s)
I20250904 07:52:57.539418 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.542730 1834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: Bootstrap starting.
I20250904 07:52:57.543855 1834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.545174 1834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: No bootstrap required, opened a new log
I20250904 07:52:57.545684 1834 raft_consensus.cc:357] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER }
I20250904 07:52:57.545799 1834 raft_consensus.cc:383] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.545866 1834 raft_consensus.cc:738] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf46dde08ed84084819ee97e5b15916b, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.546052 1834 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER }
I20250904 07:52:57.546177 1834 raft_consensus.cc:397] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.546231 1834 raft_consensus.cc:491] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.546303 1834 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.547771 1834 raft_consensus.cc:513] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER }
I20250904 07:52:57.547956 1834 leader_election.cc:304] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf46dde08ed84084819ee97e5b15916b; no voters:
I20250904 07:52:57.548280 1834 leader_election.cc:290] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.548393 1837 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.548892 1837 raft_consensus.cc:695] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 1 LEADER]: Becoming Leader. State: Replica: cf46dde08ed84084819ee97e5b15916b, State: Running, Role: LEADER
I20250904 07:52:57.549139 1834 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:57.549158 1837 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER }
I20250904 07:52:57.550050 1838 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf46dde08ed84084819ee97e5b15916b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER } }
I20250904 07:52:57.550122 1839 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf46dde08ed84084819ee97e5b15916b. Latest consensus state: current_term: 1 leader_uuid: "cf46dde08ed84084819ee97e5b15916b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf46dde08ed84084819ee97e5b15916b" member_type: VOTER } }
I20250904 07:52:57.550324 1839 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:57.550366 1838 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:57.553565 1842 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:57.554736 1842 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:57.555099 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:57.557365 1842 catalog_manager.cc:1349] Generated new cluster ID: cb690464d3bd4ec09154ef894ed63038
I20250904 07:52:57.557454 1842 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:57.566458 1842 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:57.567134 1842 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:57.572780 1842 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: Generated new TSK 0
I20250904 07:52:57.572992 1842 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:57.587432 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:57.590351 1855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.590838 1856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.590984 1858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:57.591360 413 server_base.cc:1047] running on GCE node
I20250904 07:52:57.591607 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:57.591670 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:57.591744 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972377591710 us; error 0 us; skew 500 ppm
I20250904 07:52:57.591907 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:57.592917 413 webserver.cc:480] Webserver started at http://127.0.103.65:46821/ using document root <none> and password file <none>
I20250904 07:52:57.593112 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:57.593197 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:57.593317 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:57.593842 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "a6a4e7a1521944e5b3410503d0ef4aa5"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.595930 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:52:57.597311 1863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.597630 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:57.597736 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "a6a4e7a1521944e5b3410503d0ef4aa5"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.597846 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:57.613471 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:57.613974 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:57.616035 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:57.616166 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.616289 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:57.616386 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.639103 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41515
I20250904 07:52:57.639146 1933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41515 every 8 connection(s)
I20250904 07:52:57.640739 1934 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37245
I20250904 07:52:57.640885 1934 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:57.641281 1934 heartbeater.cc:507] Master 127.0.103.126:37245 requested a full tablet report, sending...
I20250904 07:52:57.642038 1789 ts_manager.cc:194] Registered new tserver with Master: a6a4e7a1521944e5b3410503d0ef4aa5 (127.0.103.65:41515)
I20250904 07:52:57.642266 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002278814s
I20250904 07:52:57.643172 1789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36788
I20250904 07:52:57.652793 1789 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36808:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:57.654367 1934 heartbeater.cc:499] Master 127.0.103.126:37245 was elected leader, sending a full tablet report...
I20250904 07:52:57.665460 1894 tablet_service.cc:1468] Processing CreateTablet for tablet 5e9210d5fcbb44f89b18f4b9d8744f25 (DEFAULT_TABLE table=client-testtb [id=f9e0c3014b314d9c9b5725798fa44698]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:57.665416 1893 tablet_service.cc:1468] Processing CreateTablet for tablet 014fce8e33f2457694a7b575ba244613 (DEFAULT_TABLE table=client-testtb [id=f9e0c3014b314d9c9b5725798fa44698]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:57.666100 1894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9210d5fcbb44f89b18f4b9d8744f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.666306 1893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014fce8e33f2457694a7b575ba244613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.672899 1944 tablet_bootstrap.cc:492] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: Bootstrap starting.
I20250904 07:52:57.674338 1944 tablet_bootstrap.cc:654] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.675768 1944 tablet_bootstrap.cc:492] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: No bootstrap required, opened a new log
I20250904 07:52:57.675868 1944 ts_tablet_manager.cc:1397] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:52:57.676384 1944 raft_consensus.cc:357] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.676510 1944 raft_consensus.cc:383] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.676571 1944 raft_consensus.cc:738] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4e7a1521944e5b3410503d0ef4aa5, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.676753 1944 consensus_queue.cc:260] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.676918 1944 raft_consensus.cc:397] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.676985 1944 raft_consensus.cc:491] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.677089 1944 raft_consensus.cc:3058] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.679556 1944 raft_consensus.cc:513] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.679751 1944 leader_election.cc:304] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a4e7a1521944e5b3410503d0ef4aa5; no voters:
I20250904 07:52:57.680195 1944 leader_election.cc:290] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.680367 1946 raft_consensus.cc:2802] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.680828 1944 ts_tablet_manager.cc:1428] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20250904 07:52:57.681085 1946 raft_consensus.cc:695] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a4e7a1521944e5b3410503d0ef4aa5, State: Running, Role: LEADER
I20250904 07:52:57.681308 1944 tablet_bootstrap.cc:492] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: Bootstrap starting.
I20250904 07:52:57.681375 1946 consensus_queue.cc:237] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.682968 1944 tablet_bootstrap.cc:654] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.683590 1789 catalog_manager.cc:5582] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a4e7a1521944e5b3410503d0ef4aa5 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:57.685496 1944 tablet_bootstrap.cc:492] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: No bootstrap required, opened a new log
I20250904 07:52:57.685645 1944 ts_tablet_manager.cc:1397] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:52:57.686327 1944 raft_consensus.cc:357] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.686483 1944 raft_consensus.cc:383] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.686570 1944 raft_consensus.cc:738] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4e7a1521944e5b3410503d0ef4aa5, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.686818 1944 consensus_queue.cc:260] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.686945 1944 raft_consensus.cc:397] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.687045 1944 raft_consensus.cc:491] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.687139 1944 raft_consensus.cc:3058] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.689363 1944 raft_consensus.cc:513] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.689543 1944 leader_election.cc:304] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a4e7a1521944e5b3410503d0ef4aa5; no voters:
I20250904 07:52:57.689744 1944 leader_election.cc:290] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.689813 1946 raft_consensus.cc:2802] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.689985 1946 raft_consensus.cc:695] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 LEADER]: Becoming Leader. State: Replica: a6a4e7a1521944e5b3410503d0ef4aa5, State: Running, Role: LEADER
I20250904 07:52:57.690207 1944 ts_tablet_manager.cc:1428] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:52:57.690208 1946 consensus_queue.cc:237] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } }
I20250904 07:52:57.692031 1789 catalog_manager.cc:5582] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a4e7a1521944e5b3410503d0ef4aa5 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4e7a1521944e5b3410503d0ef4aa5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41515 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:57.702157 1789 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36808:
table { table_name: "client-testtb" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "non_null_with_default" remove_default: true } } } modify_external_catalogs: true
I20250904 07:52:57.707703 1954 tablet.cc:1722] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250904 07:52:57.708379 1955 tablet.cc:1722] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250904 07:52:57.709743 1955 tablet.cc:1620] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: MemRowSet was empty: no flush needed.
I20250904 07:52:57.709743 1954 tablet.cc:1620] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: MemRowSet was empty: no flush needed.
I20250904 07:52:57.718240 1789 catalog_manager.cc:6127] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: client-testtb [id=f9e0c3014b314d9c9b5725798fa44698] alter complete (version 1)
I20250904 07:52:57.737325 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:57.744997 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:57.745272 413 tablet_replica.cc:331] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5: stopping tablet replica
I20250904 07:52:57.745483 413 raft_consensus.cc:2241] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.745635 413 raft_consensus.cc:2270] T 5e9210d5fcbb44f89b18f4b9d8744f25 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.746332 413 tablet_replica.cc:331] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5: stopping tablet replica
I20250904 07:52:57.746435 413 raft_consensus.cc:2241] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.746574 413 raft_consensus.cc:2270] T 014fce8e33f2457694a7b575ba244613 P a6a4e7a1521944e5b3410503d0ef4aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.760771 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:57.772150 413 master.cc:561] Master@127.0.103.126:37245 shutting down...
I20250904 07:52:57.780134 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:57.780308 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:57.780460 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P cf46dde08ed84084819ee97e5b15916b: stopping tablet replica
I20250904 07:52:57.784188 413 master.cc:583] Master@127.0.103.126:37245 shutdown complete.
[ OK ] ClientTest.TestBatchWithPartialErrorOfMissingRequiredColumn (307 ms)
[ RUN ] ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered
I20250904 07:52:57.792004 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:43929
I20250904 07:52:57.792428 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:57.794431 1957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.794705 1958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.795112 1960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:57.795969 413 server_base.cc:1047] running on GCE node
I20250904 07:52:57.796176 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:57.796222 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:57.796255 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972377796253 us; error 0 us; skew 500 ppm
I20250904 07:52:57.796396 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:57.797477 413 webserver.cc:480] Webserver started at http://127.0.103.126:45229/ using document root <none> and password file <none>
I20250904 07:52:57.797650 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:57.797705 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:57.797812 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:57.798314 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "1f9e456794624ea6862116ee39a9827c"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.800395 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:57.801754 1965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.802109 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:52:57.802193 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "1f9e456794624ea6862116ee39a9827c"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.802284 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:57.832973 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:57.833381 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:57.845551 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:43929
I20250904 07:52:57.845570 2026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:43929 every 8 connection(s)
I20250904 07:52:57.846688 2027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.850209 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c: Bootstrap starting.
I20250904 07:52:57.851244 2027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:57.852480 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c: No bootstrap required, opened a new log
I20250904 07:52:57.853057 2027 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER }
I20250904 07:52:57.853168 2027 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:57.853204 2027 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9e456794624ea6862116ee39a9827c, State: Initialized, Role: FOLLOWER
I20250904 07:52:57.853417 2027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER }
I20250904 07:52:57.853519 2027 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:57.853574 2027 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:57.853646 2027 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:57.855201 2027 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER }
I20250904 07:52:57.855350 2027 leader_election.cc:304] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f9e456794624ea6862116ee39a9827c; no voters:
I20250904 07:52:57.855773 2027 leader_election.cc:290] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:57.855894 2030 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:57.856321 2030 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 1 LEADER]: Becoming Leader. State: Replica: 1f9e456794624ea6862116ee39a9827c, State: Running, Role: LEADER
I20250904 07:52:57.856513 2027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:57.856529 2030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER }
I20250904 07:52:57.857453 2031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f9e456794624ea6862116ee39a9827c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER } }
I20250904 07:52:57.857474 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f9e456794624ea6862116ee39a9827c. Latest consensus state: current_term: 1 leader_uuid: "1f9e456794624ea6862116ee39a9827c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9e456794624ea6862116ee39a9827c" member_type: VOTER } }
I20250904 07:52:57.857700 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:57.858258 2031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:57.859135 2035 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:57.860746 2035 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:57.861097 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:57.863246 2035 catalog_manager.cc:1349] Generated new cluster ID: 35bd4806f79b437db3791b1eba405fed
I20250904 07:52:57.863310 2035 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:57.886497 2035 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:57.887146 2035 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:57.893649 2035 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c: Generated new TSK 0
I20250904 07:52:57.893831 2035 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:57.925658 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:57.927927 2048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.928310 2049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:57.928673 2051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:57.929037 413 server_base.cc:1047] running on GCE node
I20250904 07:52:57.929338 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:57.929394 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:57.929421 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972377929416 us; error 0 us; skew 500 ppm
I20250904 07:52:57.929543 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:57.930572 413 webserver.cc:480] Webserver started at http://127.0.103.65:36953/ using document root <none> and password file <none>
I20250904 07:52:57.930766 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:57.930827 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:57.930939 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:57.931463 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "793026a4477944808b8373c24d020d10"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.933683 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:57.935034 2056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.935307 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:57.935402 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "793026a4477944808b8373c24d020d10"
format_stamp: "Formatted at 2025-09-04 07:52:57 on dist-test-slave-ghlz"
I20250904 07:52:57.935489 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:57.945925 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:57.946359 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:57.947769 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:57.947827 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.947939 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:57.948045 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:57.971464 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45159
I20250904 07:52:57.971491 2126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45159 every 8 connection(s)
I20250904 07:52:57.973345 2127 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43929
I20250904 07:52:57.973445 2127 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:57.973731 2127 heartbeater.cc:507] Master 127.0.103.126:43929 requested a full tablet report, sending...
I20250904 07:52:57.974414 1982 ts_manager.cc:194] Registered new tserver with Master: 793026a4477944808b8373c24d020d10 (127.0.103.65:45159)
I20250904 07:52:57.974983 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002253113s
I20250904 07:52:57.975353 1982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42426
I20250904 07:52:57.983667 1982 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42432:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:57.986349 2127 heartbeater.cc:499] Master 127.0.103.126:43929 was elected leader, sending a full tablet report...
I20250904 07:52:57.997568 2091 tablet_service.cc:1468] Processing CreateTablet for tablet 2d03e70563f64ae7a55842bf968a09d7 (DEFAULT_TABLE table=client-testtb [id=3bee722ce99b4a09a45f2dbef1ab59e5]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:57.997568 2090 tablet_service.cc:1468] Processing CreateTablet for tablet 3d92874eef094d43a9211e7216fc2516 (DEFAULT_TABLE table=client-testtb [id=3bee722ce99b4a09a45f2dbef1ab59e5]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:57.998121 2091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d03e70563f64ae7a55842bf968a09d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:57.998297 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d92874eef094d43a9211e7216fc2516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.002604 2137 tablet_bootstrap.cc:492] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: Bootstrap starting.
I20250904 07:52:58.004014 2137 tablet_bootstrap.cc:654] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.005970 2137 tablet_bootstrap.cc:492] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: No bootstrap required, opened a new log
I20250904 07:52:58.006069 2137 ts_tablet_manager.cc:1397] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:52:58.006565 2137 raft_consensus.cc:357] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.006688 2137 raft_consensus.cc:383] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.006738 2137 raft_consensus.cc:738] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793026a4477944808b8373c24d020d10, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.006908 2137 consensus_queue.cc:260] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.007009 2137 raft_consensus.cc:397] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:58.007064 2137 raft_consensus.cc:491] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:58.007129 2137 raft_consensus.cc:3058] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.008971 2137 raft_consensus.cc:513] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.009146 2137 leader_election.cc:304] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793026a4477944808b8373c24d020d10; no voters:
I20250904 07:52:58.009510 2137 leader_election.cc:290] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:58.009603 2139 raft_consensus.cc:2802] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.010114 2137 ts_tablet_manager.cc:1428] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250904 07:52:58.010370 2137 tablet_bootstrap.cc:492] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: Bootstrap starting.
I20250904 07:52:58.010795 2139 raft_consensus.cc:695] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 1 LEADER]: Becoming Leader. State: Replica: 793026a4477944808b8373c24d020d10, State: Running, Role: LEADER
I20250904 07:52:58.011060 2139 consensus_queue.cc:237] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.011636 2137 tablet_bootstrap.cc:654] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.013430 2137 tablet_bootstrap.cc:492] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: No bootstrap required, opened a new log
I20250904 07:52:58.013598 2137 ts_tablet_manager.cc:1397] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20250904 07:52:58.013559 1981 catalog_manager.cc:5582] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793026a4477944808b8373c24d020d10 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "793026a4477944808b8373c24d020d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:58.014276 2137 raft_consensus.cc:357] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.014397 2137 raft_consensus.cc:383] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.014443 2137 raft_consensus.cc:738] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793026a4477944808b8373c24d020d10, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.014606 2137 consensus_queue.cc:260] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.014793 2137 raft_consensus.cc:397] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:58.014899 2137 raft_consensus.cc:491] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:58.015036 2137 raft_consensus.cc:3058] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.017212 2137 raft_consensus.cc:513] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.017410 2137 leader_election.cc:304] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793026a4477944808b8373c24d020d10; no voters:
I20250904 07:52:58.017570 2137 leader_election.cc:290] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:58.017632 2139 raft_consensus.cc:2802] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.017817 2139 raft_consensus.cc:695] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 1 LEADER]: Becoming Leader. State: Replica: 793026a4477944808b8373c24d020d10, State: Running, Role: LEADER
I20250904 07:52:58.017956 2137 ts_tablet_manager.cc:1428] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250904 07:52:58.018050 2139 consensus_queue.cc:237] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } }
I20250904 07:52:58.025267 1981 catalog_manager.cc:5582] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793026a4477944808b8373c24d020d10 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "793026a4477944808b8373c24d020d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793026a4477944808b8373c24d020d10" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45159 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:58.052896 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:52:58.063205 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:52:58.063496 413 tablet_replica.cc:331] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10: stopping tablet replica
I20250904 07:52:58.063709 413 raft_consensus.cc:2241] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:58.063982 413 raft_consensus.cc:2270] T 2d03e70563f64ae7a55842bf968a09d7 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:58.064811 413 tablet_replica.cc:331] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10: stopping tablet replica
I20250904 07:52:58.064967 413 raft_consensus.cc:2241] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:58.065094 413 raft_consensus.cc:2270] T 3d92874eef094d43a9211e7216fc2516 P 793026a4477944808b8373c24d020d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:58.079097 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:52:58.083194 413 master.cc:561] Master@127.0.103.126:43929 shutting down...
I20250904 07:52:58.097116 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:52:58.097314 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:52:58.097465 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1f9e456794624ea6862116ee39a9827c: stopping tablet replica
I20250904 07:52:58.110853 413 master.cc:583] Master@127.0.103.126:43929 shutdown complete.
[ OK ] ClientTest.TestApplyToSessionWithoutFlushing_OpsBuffered (327 ms)
[ RUN ] ClientTest.TestAutoFlushBackgroundAndErrorCollector
I20250904 07:52:58.119535 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:38915
I20250904 07:52:58.119978 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:58.122017 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.122402 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.122611 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:58.122707 413 server_base.cc:1047] running on GCE node
I20250904 07:52:58.123003 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:58.123052 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:58.123073 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972378123071 us; error 0 us; skew 500 ppm
I20250904 07:52:58.123176 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:58.124181 413 webserver.cc:480] Webserver started at http://127.0.103.126:43867/ using document root <none> and password file <none>
I20250904 07:52:58.124336 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:58.124382 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:58.124459 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:58.124997 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "7f4f1af1682c402d8305da8c5d5f1da7"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.127027 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:52:58.128394 2155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.128736 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:58.128819 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "7f4f1af1682c402d8305da8c5d5f1da7"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.128943 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:58.150213 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:58.150688 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:58.162977 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:38915
I20250904 07:52:58.162999 2216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:38915 every 8 connection(s)
I20250904 07:52:58.164098 2217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.167795 2217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7: Bootstrap starting.
I20250904 07:52:58.168913 2217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.170228 2217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7: No bootstrap required, opened a new log
I20250904 07:52:58.170789 2217 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER }
I20250904 07:52:58.170918 2217 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.171038 2217 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4f1af1682c402d8305da8c5d5f1da7, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.171247 2217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER }
I20250904 07:52:58.171386 2217 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:58.171465 2217 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:58.171578 2217 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.173242 2217 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER }
I20250904 07:52:58.173452 2217 leader_election.cc:304] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f4f1af1682c402d8305da8c5d5f1da7; no voters:
I20250904 07:52:58.173815 2217 leader_election.cc:290] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:58.173918 2220 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.174362 2220 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 1 LEADER]: Becoming Leader. State: Replica: 7f4f1af1682c402d8305da8c5d5f1da7, State: Running, Role: LEADER
I20250904 07:52:58.174597 2220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER }
I20250904 07:52:58.174638 2217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:52:58.175319 2222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4f1af1682c402d8305da8c5d5f1da7. Latest consensus state: current_term: 1 leader_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER } }
I20250904 07:52:58.175376 2221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4f1af1682c402d8305da8c5d5f1da7" member_type: VOTER } }
I20250904 07:52:58.175489 2222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:58.175498 2221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:52:58.176615 2225 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:52:58.177809 2225 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:52:58.180532 2225 catalog_manager.cc:1349] Generated new cluster ID: 36c40339aa0a45eea84546ca2ea74b46
I20250904 07:52:58.180636 2225 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:52:58.181274 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:52:58.197360 2225 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:52:58.198040 2225 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:52:58.206027 2225 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7: Generated new TSK 0
I20250904 07:52:58.206231 2225 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:52:58.213649 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:58.215929 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.216638 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:58.216745 413 server_base.cc:1047] running on GCE node
W20250904 07:52:58.216774 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:58.217218 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:58.217290 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:58.217356 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972378217337 us; error 0 us; skew 500 ppm
I20250904 07:52:58.217526 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:58.218559 413 webserver.cc:480] Webserver started at http://127.0.103.65:35225/ using document root <none> and password file <none>
I20250904 07:52:58.218797 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:58.218880 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:58.219023 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:58.219559 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "52f35e2e598e4b4390f48a4b9888d106"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.221750 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:58.223151 2246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.223451 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:58.223589 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "52f35e2e598e4b4390f48a4b9888d106"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.223712 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:58.231838 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:58.232223 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:58.233675 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:58.233824 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.233986 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:58.234102 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.249696 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:46697
I20250904 07:52:58.250041 2316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:46697 every 8 connection(s)
I20250904 07:52:58.251135 2317 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38915
I20250904 07:52:58.251241 2317 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:58.251524 2317 heartbeater.cc:507] Master 127.0.103.126:38915 requested a full tablet report, sending...
I20250904 07:52:58.252241 2172 ts_manager.cc:194] Registered new tserver with Master: 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:52:58.252817 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002312348s
I20250904 07:52:58.253161 2172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59612
I20250904 07:52:58.261755 2172 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:52:58.264221 2317 heartbeater.cc:499] Master 127.0.103.126:38915 was elected leader, sending a full tablet report...
I20250904 07:52:58.272706 2280 tablet_service.cc:1468] Processing CreateTablet for tablet d207638da3bf43fc96e29e53cbd76fd4 (DEFAULT_TABLE table=client-testtb [id=1126a38754b746ebaec17a7ef780fbed]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:52:58.272815 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 2196fc4e937340e99876b54644e94905 (DEFAULT_TABLE table=client-testtb [id=1126a38754b746ebaec17a7ef780fbed]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:52:58.273098 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d207638da3bf43fc96e29e53cbd76fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.273391 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2196fc4e937340e99876b54644e94905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.278329 2327 tablet_bootstrap.cc:492] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:58.279516 2327 tablet_bootstrap.cc:654] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.281029 2327 tablet_bootstrap.cc:492] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:58.281183 2327 ts_tablet_manager.cc:1397] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:52:58.281872 2327 raft_consensus.cc:357] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.282003 2327 raft_consensus.cc:383] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.282063 2327 raft_consensus.cc:738] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.282233 2327 consensus_queue.cc:260] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.282361 2327 raft_consensus.cc:397] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:58.282418 2327 raft_consensus.cc:491] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:58.282491 2327 raft_consensus.cc:3058] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.284065 2327 raft_consensus.cc:513] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.284233 2327 leader_election.cc:304] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106; no voters:
I20250904 07:52:58.284579 2327 leader_election.cc:290] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:58.284735 2329 raft_consensus.cc:2802] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.285233 2329 raft_consensus.cc:695] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:52:58.285290 2327 ts_tablet_manager.cc:1428] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:52:58.285594 2327 tablet_bootstrap.cc:492] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:58.285569 2329 consensus_queue.cc:237] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.288081 2327 tablet_bootstrap.cc:654] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.288025 2171 catalog_manager.cc:5582] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:58.289728 2327 tablet_bootstrap.cc:492] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:58.289880 2327 ts_tablet_manager.cc:1397] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250904 07:52:58.290468 2327 raft_consensus.cc:357] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.290650 2327 raft_consensus.cc:383] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.290738 2327 raft_consensus.cc:738] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.290961 2327 consensus_queue.cc:260] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.291139 2327 raft_consensus.cc:397] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:52:58.291245 2327 raft_consensus.cc:491] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:52:58.291342 2327 raft_consensus.cc:3058] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.293066 2327 raft_consensus.cc:513] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.293223 2327 leader_election.cc:304] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106; no voters:
I20250904 07:52:58.293377 2327 leader_election.cc:290] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:52:58.293433 2329 raft_consensus.cc:2802] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.293562 2329 raft_consensus.cc:695] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:52:58.293754 2327 ts_tablet_manager.cc:1428] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250904 07:52:58.293785 2329 consensus_queue.cc:237] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.295159 2171 catalog_manager.cc:5582] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:52:58.301606 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:52:58.303956 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.304086 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.304517 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:58.305027 413 server_base.cc:1047] running on GCE node
I20250904 07:52:58.305312 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:58.305372 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:58.305429 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972378305399 us; error 0 us; skew 500 ppm
I20250904 07:52:58.305557 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:58.307065 413 webserver.cc:480] Webserver started at http://127.0.103.66:35073/ using document root <none> and password file <none>
I20250904 07:52:58.307233 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:58.307304 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:58.307403 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:58.307927 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-1-root/instance:
uuid: "a1047321e14f440c818a8f606e49cb81"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.310384 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:52:58.311800 2344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.312142 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:58.312249 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "a1047321e14f440c818a8f606e49cb81"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.312367 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:58.322959 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:58.323393 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:58.325167 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:58.325271 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.325379 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:58.325464 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.354308 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40289
I20250904 07:52:58.354415 2413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40289 every 8 connection(s)
I20250904 07:52:58.355916 2414 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38915
I20250904 07:52:58.355981 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250904 07:52:58.356021 2414 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:58.356397 2414 heartbeater.cc:507] Master 127.0.103.126:38915 requested a full tablet report, sending...
I20250904 07:52:58.357131 2172 ts_manager.cc:194] Registered new tserver with Master: a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:52:58.358136 2172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59640
W20250904 07:52:58.358592 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.359232 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:52:58.360092 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:52:58.360405 413 server_base.cc:1047] running on GCE node
I20250904 07:52:58.360666 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:52:58.360728 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:52:58.360805 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972378360772 us; error 0 us; skew 500 ppm
I20250904 07:52:58.360973 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:52:58.361941 413 webserver.cc:480] Webserver started at http://127.0.103.67:42567/ using document root <none> and password file <none>
I20250904 07:52:58.362125 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:52:58.362214 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:52:58.362339 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:52:58.362915 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-2-root/instance:
uuid: "7a00f455205e4b0eb7aba39391375837"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.365687 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:52:58.367244 2424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.367548 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:52:58.367646 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "7a00f455205e4b0eb7aba39391375837"
format_stamp: "Formatted at 2025-09-04 07:52:58 on dist-test-slave-ghlz"
I20250904 07:52:58.367765 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestAutoFlushBackgroundAndErrorCollector.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:52:58.369316 2414 heartbeater.cc:499] Master 127.0.103.126:38915 was elected leader, sending a full tablet report...
I20250904 07:52:58.378202 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:52:58.378626 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:52:58.380184 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:52:58.380285 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.380389 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:52:58.380474 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:52:58.410799 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:42657
I20250904 07:52:58.410857 2493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:42657 every 8 connection(s)
I20250904 07:52:58.412552 2494 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38915
I20250904 07:52:58.412689 2494 heartbeater.cc:461] Registering TS with master...
I20250904 07:52:58.413189 2494 heartbeater.cc:507] Master 127.0.103.126:38915 requested a full tablet report, sending...
I20250904 07:52:58.413884 2172 ts_manager.cc:194] Registered new tserver with Master: 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:52:58.414211 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.002345251s
I20250904 07:52:58.414685 2172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59666
I20250904 07:52:58.415086 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:52:58.416035 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:52:58.424484 2280 tablet_service.cc:1468] Processing CreateTablet for tablet cf534c94cabb4b49b5e6ee589c38501c (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:58.424575 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 14bdb1ac7adf4767bafcc8964e63e885 (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:58.425009 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf534c94cabb4b49b5e6ee589c38501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.426398 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14bdb1ac7adf4767bafcc8964e63e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.427187 2494 heartbeater.cc:499] Master 127.0.103.126:38915 was elected leader, sending a full tablet report...
I20250904 07:52:58.433560 2327 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:58.433633 2454 tablet_service.cc:1468] Processing CreateTablet for tablet cf534c94cabb4b49b5e6ee589c38501c (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:58.433701 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 14bdb1ac7adf4767bafcc8964e63e885 (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:58.434157 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf534c94cabb4b49b5e6ee589c38501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.434409 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14bdb1ac7adf4767bafcc8964e63e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.435937 2327 tablet_bootstrap.cc:654] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.443266 2379 tablet_service.cc:1468] Processing CreateTablet for tablet cf534c94cabb4b49b5e6ee589c38501c (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:58.443987 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 14bdb1ac7adf4767bafcc8964e63e885 (DEFAULT_TABLE table=table.0 [id=a97e4403b7d2456cbd68f0731ae004d9]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:58.445271 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf534c94cabb4b49b5e6ee589c38501c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.445623 2500 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:52:58.447685 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14bdb1ac7adf4767bafcc8964e63e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:58.448076 2327 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:58.448282 2327 ts_tablet_manager.cc:1397] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250904 07:52:58.448904 2500 tablet_bootstrap.cc:654] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.449178 2327 raft_consensus.cc:357] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.449504 2327 raft_consensus.cc:383] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.449702 2327 raft_consensus.cc:738] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.450067 2327 consensus_queue.cc:260] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.453552 2327 ts_tablet_manager.cc:1428] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250904 07:52:58.456321 2327 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:58.457413 2500 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:52:58.457629 2500 ts_tablet_manager.cc:1397] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20250904 07:52:58.458704 2327 tablet_bootstrap.cc:654] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.458807 2500 raft_consensus.cc:357] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.459002 2500 raft_consensus.cc:383] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.459527 2500 raft_consensus.cc:738] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.459806 2500 consensus_queue.cc:260] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.460834 2500 ts_tablet_manager.cc:1428] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:52:58.461252 2500 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:52:58.462635 2502 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:52:58.463322 2500 tablet_bootstrap.cc:654] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.465055 2502 tablet_bootstrap.cc:654] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.465862 2327 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:58.466039 2327 ts_tablet_manager.cc:1397] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20250904 07:52:58.467006 2327 raft_consensus.cc:357] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.467264 2327 raft_consensus.cc:383] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.467360 2327 raft_consensus.cc:738] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.467623 2327 consensus_queue.cc:260] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.468109 2327 ts_tablet_manager.cc:1428] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:52:58.468348 2500 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:52:58.468539 2500 ts_tablet_manager.cc:1397] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250904 07:52:58.470397 2500 raft_consensus.cc:357] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.470669 2500 raft_consensus.cc:383] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.471304 2500 raft_consensus.cc:738] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.471617 2500 consensus_queue.cc:260] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.472164 2500 ts_tablet_manager.cc:1428] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250904 07:52:58.473299 2502 tablet_bootstrap.cc:492] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:52:58.473425 2502 ts_tablet_manager.cc:1397] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250904 07:52:58.474094 2502 raft_consensus.cc:357] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.474573 2502 raft_consensus.cc:383] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.474736 2502 raft_consensus.cc:738] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.475106 2502 consensus_queue.cc:260] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.475797 2502 ts_tablet_manager.cc:1428] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:52:58.476073 2502 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:52:58.477510 2502 tablet_bootstrap.cc:654] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:58.479123 2502 tablet_bootstrap.cc:492] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:52:58.479256 2502 ts_tablet_manager.cc:1397] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:52:58.479811 2502 raft_consensus.cc:357] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.479964 2502 raft_consensus.cc:383] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:58.480036 2502 raft_consensus.cc:738] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:52:58.480266 2502 consensus_queue.cc:260] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.480707 2502 ts_tablet_manager.cc:1428] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:52:58.500820 2318 tablet.cc:2378] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:52:58.523653 2507 raft_consensus.cc:491] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:52:58.523788 2507 raft_consensus.cc:513] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.525249 2507 leader_election.cc:290] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:52:58.532584 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14bdb1ac7adf4767bafcc8964e63e885" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:52:58.532629 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14bdb1ac7adf4767bafcc8964e63e885" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:52:58.533003 2466 raft_consensus.cc:2466] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:52:58.533021 2292 raft_consensus.cc:2466] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:52:58.533502 2345 leader_election.cc:304] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:52:58.533737 2507 raft_consensus.cc:2802] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:52:58.533830 2507 raft_consensus.cc:491] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:52:58.533932 2507 raft_consensus.cc:3058] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.535713 2507 raft_consensus.cc:513] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.536643 2507 leader_election.cc:290] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:52:58.536998 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14bdb1ac7adf4767bafcc8964e63e885" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:52:58.537153 2292 raft_consensus.cc:3058] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.537148 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14bdb1ac7adf4767bafcc8964e63e885" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:52:58.537376 2466 raft_consensus.cc:3058] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.539141 2292 raft_consensus.cc:2466] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:52:58.539281 2466 raft_consensus.cc:2466] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:52:58.539803 2345 leader_election.cc:304] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:52:58.540742 2507 raft_consensus.cc:2802] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.541136 2507 raft_consensus.cc:695] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:52:58.541406 2507 consensus_queue.cc:237] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:58.544028 2172 catalog_manager.cc:5582] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:52:58.595603 2507 raft_consensus.cc:491] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:52:58.595803 2507 raft_consensus.cc:513] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.596393 2507 leader_election.cc:290] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:52:58.596685 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf534c94cabb4b49b5e6ee589c38501c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:52:58.596725 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf534c94cabb4b49b5e6ee589c38501c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:52:58.596942 2466 raft_consensus.cc:2466] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:52:58.596971 2292 raft_consensus.cc:2466] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:52:58.597323 2345 leader_election.cc:304] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:52:58.597579 2507 raft_consensus.cc:2802] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:52:58.597703 2507 raft_consensus.cc:491] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:52:58.597859 2507 raft_consensus.cc:3058] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.599771 2507 raft_consensus.cc:513] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.600265 2507 leader_election.cc:290] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:52:58.600625 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf534c94cabb4b49b5e6ee589c38501c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:52:58.600654 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf534c94cabb4b49b5e6ee589c38501c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:52:58.600806 2466 raft_consensus.cc:3058] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.600829 2292 raft_consensus.cc:3058] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:58.603453 2466 raft_consensus.cc:2466] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:52:58.603538 2292 raft_consensus.cc:2466] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:52:58.603798 2345 leader_election.cc:304] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:52:58.604101 2507 raft_consensus.cc:2802] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:58.604240 2507 raft_consensus.cc:695] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:52:58.604462 2507 consensus_queue.cc:237] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:58.606658 2172 catalog_manager.cc:5582] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:52:58.644341 2292 raft_consensus.cc:1273] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:52:58.645088 2291 raft_consensus.cc:1273] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:52:58.645135 2466 raft_consensus.cc:1273] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:52:58.645484 2510 consensus_queue.cc:1035] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:52:58.645870 2507 consensus_queue.cc:1035] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:52:58.646137 2516 consensus_queue.cc:1035] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:52:58.647603 2466 raft_consensus.cc:1273] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:52:58.648121 2516 consensus_queue.cc:1035] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:52:58.662864 2518 mvcc.cc:204] Tried to move back new op lower bound from 7196558862914277376 to 7196558862766014464. Current Snapshot: MvccSnapshot[applied={T|T < 7196558862914277376}]
W20250904 07:52:59.014482 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 279ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:52:59.267715 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 532ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:52:59.521315 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 785ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:52:59.683549 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00114e+06us
I20250904 07:52:59.684988 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:52:59.685784 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:52:59.699334 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 6a938a8ee4894764b610f1a154182a46 (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:59.699744 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a938a8ee4894764b610f1a154182a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.700845 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 6a938a8ee4894764b610f1a154182a46 (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:59.701231 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a938a8ee4894764b610f1a154182a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.701260 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 370e09c64ae047deb24be40c12777f0b (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:59.701539 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370e09c64ae047deb24be40c12777f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.707232 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 370e09c64ae047deb24be40c12777f0b (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:59.707695 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370e09c64ae047deb24be40c12777f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.708336 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 6a938a8ee4894764b610f1a154182a46 (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:52:59.708791 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a938a8ee4894764b610f1a154182a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.709118 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 370e09c64ae047deb24be40c12777f0b (DEFAULT_TABLE table=table.1 [id=69fe9310e779458ba7e4727fd2e55bab]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:52:59.709491 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370e09c64ae047deb24be40c12777f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:52:59.715812 2540 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:52:59.717242 2540 tablet_bootstrap.cc:654] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.724990 2540 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:52:59.725119 2540 ts_tablet_manager.cc:1397] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20250904 07:52:59.725862 2540 raft_consensus.cc:357] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.726035 2540 raft_consensus.cc:383] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.726096 2540 raft_consensus.cc:738] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.726308 2540 consensus_queue.cc:260] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.731508 2540 ts_tablet_manager.cc:1428] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250904 07:52:59.737202 2540 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:52:59.738593 2540 tablet_bootstrap.cc:654] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.740602 2545 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:59.741833 2541 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:52:59.742621 2545 tablet_bootstrap.cc:654] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.743240 2540 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:52:59.743357 2540 ts_tablet_manager.cc:1397] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250904 07:52:59.744056 2540 raft_consensus.cc:357] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.744228 2540 raft_consensus.cc:383] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.744287 2540 raft_consensus.cc:738] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.744490 2540 consensus_queue.cc:260] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.745062 2540 ts_tablet_manager.cc:1428] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250904 07:52:59.745730 2541 tablet_bootstrap.cc:654] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.746747 2545 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:59.746868 2545 ts_tablet_manager.cc:1397] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250904 07:52:59.747509 2545 raft_consensus.cc:357] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.747675 2545 raft_consensus.cc:383] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.747725 2545 raft_consensus.cc:738] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.747915 2545 consensus_queue.cc:260] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.758003 2545 ts_tablet_manager.cc:1428] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20250904 07:52:59.758345 2545 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:52:59.758942 2541 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:52:59.759052 2541 ts_tablet_manager.cc:1397] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20250904 07:52:59.759714 2545 tablet_bootstrap.cc:654] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.759670 2541 raft_consensus.cc:357] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.759831 2541 raft_consensus.cc:383] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.759878 2541 raft_consensus.cc:738] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.760097 2541 consensus_queue.cc:260] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.760596 2541 ts_tablet_manager.cc:1428] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:59.760818 2541 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:52:59.762244 2541 tablet_bootstrap.cc:654] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:52:59.769927 2541 tablet_bootstrap.cc:492] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:52:59.769999 2545 tablet_bootstrap.cc:492] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:52:59.770038 2541 ts_tablet_manager.cc:1397] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20250904 07:52:59.770107 2545 ts_tablet_manager.cc:1397] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20250904 07:52:59.770464 2541 raft_consensus.cc:357] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.770570 2541 raft_consensus.cc:383] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.770606 2541 raft_consensus.cc:738] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.770710 2545 raft_consensus.cc:357] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.770774 2541 consensus_queue.cc:260] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:52:59.770844 2545 raft_consensus.cc:383] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:52:59.770886 2545 raft_consensus.cc:738] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:52:59.771081 2545 consensus_queue.cc:260] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.771167 2541 ts_tablet_manager.cc:1428] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:52:59.771502 2545 ts_tablet_manager.cc:1428] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:52:59.797327 2548 raft_consensus.cc:491] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:52:59.797480 2548 raft_consensus.cc:513] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.798230 2548 leader_election.cc:290] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:52:59.820711 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a938a8ee4894764b610f1a154182a46" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:52:59.820940 2464 raft_consensus.cc:2466] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:52:59.821348 2247 leader_election.cc:304] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:52:59.821609 2548 raft_consensus.cc:2802] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:52:59.821678 2548 raft_consensus.cc:491] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:52:59.821720 2548 raft_consensus.cc:3058] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:59.822906 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a938a8ee4894764b610f1a154182a46" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:52:59.823102 2389 raft_consensus.cc:2466] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:52:59.823882 2548 raft_consensus.cc:513] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.824533 2548 leader_election.cc:290] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:52:59.824894 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a938a8ee4894764b610f1a154182a46" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:52:59.825021 2464 raft_consensus.cc:3058] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:59.825778 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a938a8ee4894764b610f1a154182a46" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:52:59.825934 2389 raft_consensus.cc:3058] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:52:59.827528 2464 raft_consensus.cc:2466] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:52:59.827891 2247 leader_election.cc:304] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:52:59.828116 2548 raft_consensus.cc:2802] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:52:59.828497 2548 raft_consensus.cc:695] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:52:59.828634 2389 raft_consensus.cc:2466] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:52:59.828775 2548 consensus_queue.cc:237] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:52:59.831254 2170 catalog_manager.cc:5582] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:00.009892 2507 raft_consensus.cc:491] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:00.010059 2507 raft_consensus.cc:513] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:00.010604 2507 leader_election.cc:290] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:00.010942 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "370e09c64ae047deb24be40c12777f0b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:00.011116 2464 raft_consensus.cc:2466] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:00.011070 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "370e09c64ae047deb24be40c12777f0b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:00.011296 2292 raft_consensus.cc:2466] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:00.011444 2345 leader_election.cc:304] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:00.011723 2507 raft_consensus.cc:2802] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:00.011803 2507 raft_consensus.cc:491] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:00.011886 2507 raft_consensus.cc:3058] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:00.013702 2507 raft_consensus.cc:513] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:00.014132 2507 leader_election.cc:290] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:00.014452 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "370e09c64ae047deb24be40c12777f0b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:00.014571 2464 raft_consensus.cc:3058] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:00.014513 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "370e09c64ae047deb24be40c12777f0b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:00.014734 2292 raft_consensus.cc:3058] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:00.016371 2464 raft_consensus.cc:2466] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:00.016578 2292 raft_consensus.cc:2466] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:00.016919 2345 leader_election.cc:304] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:00.017105 2507 raft_consensus.cc:2802] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:00.017472 2507 raft_consensus.cc:695] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:00.017761 2507 consensus_queue.cc:237] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:00.020253 2171 catalog_manager.cc:5582] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:00.096060 2292 raft_consensus.cc:1273] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:00.096187 2464 raft_consensus.cc:1273] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:00.096724 2507 consensus_queue.cc:1035] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:00.096994 2568 consensus_queue.cc:1035] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:00.117712 2463 raft_consensus.cc:1273] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:00.118197 2548 consensus_queue.cc:1035] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:00.118237 2389 raft_consensus.cc:1273] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:00.119405 2548 consensus_queue.cc:1035] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:00.371573 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 202ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:00.620517 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 451ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:00.869982 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 700ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:01.121744 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.0009e+06us
I20250904 07:53:01.122833 2172 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:01.123646 2172 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:01.132160 2378 tablet_service.cc:1468] Processing CreateTablet for tablet ce4fa00e235c4bca9188e975c48f9425 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:01.132269 2453 tablet_service.cc:1468] Processing CreateTablet for tablet ce4fa00e235c4bca9188e975c48f9425 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:01.132633 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fa00e235c4bca9188e975c48f9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.132711 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fa00e235c4bca9188e975c48f9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.133466 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 56252ec8672745adaa3bf8af60e68f82 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:01.133833 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56252ec8672745adaa3bf8af60e68f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.135494 2280 tablet_service.cc:1468] Processing CreateTablet for tablet ce4fa00e235c4bca9188e975c48f9425 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:01.135877 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fa00e235c4bca9188e975c48f9425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.137367 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 56252ec8672745adaa3bf8af60e68f82 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:01.137765 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56252ec8672745adaa3bf8af60e68f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.140688 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 56252ec8672745adaa3bf8af60e68f82 (DEFAULT_TABLE table=table.2 [id=27fc24fde90d44c38c497a8d22ec127d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:01.141077 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56252ec8672745adaa3bf8af60e68f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:01.145363 2589 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:01.146549 2589 tablet_bootstrap.cc:654] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.151216 2590 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:01.151598 2589 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:01.151791 2589 ts_tablet_manager.cc:1397] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250904 07:53:01.152626 2589 raft_consensus.cc:357] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.152956 2589 raft_consensus.cc:383] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.153040 2590 tablet_bootstrap.cc:654] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.153131 2589 raft_consensus.cc:738] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.153416 2589 consensus_queue.cc:260] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.157670 2594 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:01.159533 2589 ts_tablet_manager.cc:1428] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20250904 07:53:01.160089 2589 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:01.160619 2594 tablet_bootstrap.cc:654] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.161366 2589 tablet_bootstrap.cc:654] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.161427 2590 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:01.162298 2590 ts_tablet_manager.cc:1397] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250904 07:53:01.163801 2590 raft_consensus.cc:357] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.164641 2590 raft_consensus.cc:383] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.164882 2590 raft_consensus.cc:738] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.165246 2590 consensus_queue.cc:260] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.166028 2590 ts_tablet_manager.cc:1428] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250904 07:53:01.166460 2590 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:01.166824 2589 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:01.167038 2589 ts_tablet_manager.cc:1397] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250904 07:53:01.167848 2589 raft_consensus.cc:357] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.168128 2589 raft_consensus.cc:383] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.168265 2589 raft_consensus.cc:738] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.168434 2594 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:01.168517 2590 tablet_bootstrap.cc:654] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.168694 2594 ts_tablet_manager.cc:1397] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20250904 07:53:01.168624 2589 consensus_queue.cc:260] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.169221 2589 ts_tablet_manager.cc:1428] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:01.169657 2594 raft_consensus.cc:357] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.169983 2594 raft_consensus.cc:383] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.170140 2594 raft_consensus.cc:738] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.170362 2594 consensus_queue.cc:260] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.171252 2594 ts_tablet_manager.cc:1428] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:01.171658 2594 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:01.173033 2590 tablet_bootstrap.cc:492] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:01.173287 2590 ts_tablet_manager.cc:1397] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20250904 07:53:01.173369 2594 tablet_bootstrap.cc:654] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:01.174057 2590 raft_consensus.cc:357] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.174366 2590 raft_consensus.cc:383] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.174517 2590 raft_consensus.cc:738] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.174952 2590 consensus_queue.cc:260] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.175695 2590 ts_tablet_manager.cc:1428] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:01.175981 2594 tablet_bootstrap.cc:492] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:01.176193 2594 ts_tablet_manager.cc:1397] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:53:01.176947 2594 raft_consensus.cc:357] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.177166 2594 raft_consensus.cc:383] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:01.177265 2594 raft_consensus.cc:738] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:01.177486 2594 consensus_queue.cc:260] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.177893 2594 ts_tablet_manager.cc:1428] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:01.334228 2556 raft_consensus.cc:491] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:01.334405 2556 raft_consensus.cc:513] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.334944 2556 leader_election.cc:290] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:01.335309 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56252ec8672745adaa3bf8af60e68f82" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:01.335305 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56252ec8672745adaa3bf8af60e68f82" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:01.335528 2463 raft_consensus.cc:2466] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:01.335531 2389 raft_consensus.cc:2466] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:01.335976 2249 leader_election.cc:304] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:01.336257 2556 raft_consensus.cc:2802] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:01.336361 2556 raft_consensus.cc:491] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:01.336460 2556 raft_consensus.cc:3058] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.339098 2556 raft_consensus.cc:513] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.339617 2556 leader_election.cc:290] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:01.339912 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56252ec8672745adaa3bf8af60e68f82" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:01.339947 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56252ec8672745adaa3bf8af60e68f82" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:01.340049 2389 raft_consensus.cc:3058] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.340121 2463 raft_consensus.cc:3058] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.341953 2389 raft_consensus.cc:2466] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:01.342094 2463 raft_consensus.cc:2466] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:01.342427 2249 leader_election.cc:304] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:01.342684 2556 raft_consensus.cc:2802] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:01.343043 2556 raft_consensus.cc:695] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:01.343345 2556 consensus_queue.cc:237] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:01.345770 2172 catalog_manager.cc:5582] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:01.383203 2579 raft_consensus.cc:491] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:01.383374 2579 raft_consensus.cc:513] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.383953 2579 leader_election.cc:290] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:01.384281 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce4fa00e235c4bca9188e975c48f9425" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:01.384303 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce4fa00e235c4bca9188e975c48f9425" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:01.384505 2292 raft_consensus.cc:2466] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:01.384548 2463 raft_consensus.cc:2466] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:01.384920 2345 leader_election.cc:304] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:01.385150 2579 raft_consensus.cc:2802] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:01.385264 2579 raft_consensus.cc:491] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:01.385352 2579 raft_consensus.cc:3058] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.387046 2579 raft_consensus.cc:513] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.387501 2579 leader_election.cc:290] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:01.387813 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce4fa00e235c4bca9188e975c48f9425" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:01.387902 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce4fa00e235c4bca9188e975c48f9425" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:01.387962 2463 raft_consensus.cc:3058] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.388113 2292 raft_consensus.cc:3058] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:01.390235 2463 raft_consensus.cc:2466] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:01.390332 2292 raft_consensus.cc:2466] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:01.390631 2345 leader_election.cc:304] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:01.390915 2579 raft_consensus.cc:2802] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:01.391348 2579 raft_consensus.cc:695] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:01.391671 2579 consensus_queue.cc:237] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:01.394172 2172 catalog_manager.cc:5582] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:01.453428 2463 raft_consensus.cc:1273] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:01.453509 2389 raft_consensus.cc:1273] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:01.453963 2556 consensus_queue.cc:1035] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:01.454082 2292 raft_consensus.cc:1273] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:01.454353 2599 consensus_queue.cc:1035] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:01.454564 2463 raft_consensus.cc:1273] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:01.454612 2601 consensus_queue.cc:1035] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:01.455211 2601 consensus_queue.cc:1035] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:01.772569 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 252ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:02.023906 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 501ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:02.274631 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 752ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:02.472353 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00079e+06us
I20250904 07:53:02.473156 2172 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:02.473858 2172 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:02.482030 2379 tablet_service.cc:1468] Processing CreateTablet for tablet cdd1e744f16b407980acc76a77789727 (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:02.482347 2278 tablet_service.cc:1468] Processing CreateTablet for tablet cdd1e744f16b407980acc76a77789727 (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:02.482506 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd1e744f16b407980acc76a77789727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.482723 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd1e744f16b407980acc76a77789727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.483322 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 1b6c13fce2c34de39f172711a420daab (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:02.483687 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b6c13fce2c34de39f172711a420daab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.487073 2624 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:02.487181 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 1b6c13fce2c34de39f172711a420daab (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:02.487498 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b6c13fce2c34de39f172711a420daab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.488453 2624 tablet_bootstrap.cc:654] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.488896 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 1b6c13fce2c34de39f172711a420daab (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:02.489253 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b6c13fce2c34de39f172711a420daab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.482030 2453 tablet_service.cc:1468] Processing CreateTablet for tablet cdd1e744f16b407980acc76a77789727 (DEFAULT_TABLE table=table.3 [id=d952e65431dc469d82336a8ced9d710e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:02.489689 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd1e744f16b407980acc76a77789727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:02.500329 2625 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:02.501899 2625 tablet_bootstrap.cc:654] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.503638 2627 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:02.504069 2625 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:02.504215 2625 ts_tablet_manager.cc:1397] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:02.504743 2625 raft_consensus.cc:357] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.504907 2625 raft_consensus.cc:383] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.504992 2625 raft_consensus.cc:738] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.504048 2624 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:02.505190 2625 consensus_queue.cc:260] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.505247 2624 ts_tablet_manager.cc:1397] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20250904 07:53:02.505321 2627 tablet_bootstrap.cc:654] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.505733 2625 ts_tablet_manager.cc:1428] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250904 07:53:02.506001 2625 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:02.506956 2627 tablet_bootstrap.cc:492] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:02.507120 2627 ts_tablet_manager.cc:1397] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:02.507268 2624 raft_consensus.cc:357] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.507493 2624 raft_consensus.cc:383] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.508158 2624 raft_consensus.cc:738] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.508339 2624 consensus_queue.cc:260] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.508088 2625 tablet_bootstrap.cc:654] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.508504 2627 raft_consensus.cc:357] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.508682 2627 raft_consensus.cc:383] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.508807 2627 raft_consensus.cc:738] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.509122 2627 consensus_queue.cc:260] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.510341 2624 ts_tablet_manager.cc:1428] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250904 07:53:02.510356 2627 ts_tablet_manager.cc:1428] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:02.510591 2624 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:02.511111 2627 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:02.512097 2624 tablet_bootstrap.cc:654] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.512677 2627 tablet_bootstrap.cc:654] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:02.513615 2625 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:02.513896 2625 ts_tablet_manager.cc:1397] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20250904 07:53:02.514617 2625 raft_consensus.cc:357] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.514796 2625 raft_consensus.cc:383] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.514844 2627 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:02.514863 2625 raft_consensus.cc:738] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.514935 2627 ts_tablet_manager.cc:1397] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250904 07:53:02.515030 2625 consensus_queue.cc:260] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.515416 2625 ts_tablet_manager.cc:1428] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:02.515563 2627 raft_consensus.cc:357] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.515759 2627 raft_consensus.cc:383] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.515866 2627 raft_consensus.cc:738] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.516088 2627 consensus_queue.cc:260] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.516302 2624 tablet_bootstrap.cc:492] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:02.516494 2624 ts_tablet_manager.cc:1397] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20250904 07:53:02.516535 2627 ts_tablet_manager.cc:1428] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:02.517145 2624 raft_consensus.cc:357] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.517324 2624 raft_consensus.cc:383] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:02.517402 2624 raft_consensus.cc:738] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:02.517581 2624 consensus_queue.cc:260] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.517931 2624 ts_tablet_manager.cc:1428] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:02.548321 2612 raft_consensus.cc:491] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:02.548455 2612 raft_consensus.cc:513] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.549046 2612 leader_election.cc:290] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:02.549397 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b6c13fce2c34de39f172711a420daab" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:02.549453 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b6c13fce2c34de39f172711a420daab" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:02.549561 2389 raft_consensus.cc:2466] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:02.549589 2463 raft_consensus.cc:2466] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:02.549908 2249 leader_election.cc:304] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:02.550112 2612 raft_consensus.cc:2802] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:02.550189 2612 raft_consensus.cc:491] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:02.550268 2612 raft_consensus.cc:3058] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.551924 2612 raft_consensus.cc:513] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.552349 2612 leader_election.cc:290] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:02.552706 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b6c13fce2c34de39f172711a420daab" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:02.552739 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b6c13fce2c34de39f172711a420daab" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:02.552884 2389 raft_consensus.cc:3058] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.552888 2463 raft_consensus.cc:3058] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.555528 2389 raft_consensus.cc:2466] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:02.555974 2249 leader_election.cc:304] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:02.556171 2612 raft_consensus.cc:2802] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:02.556311 2463 raft_consensus.cc:2466] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:02.556574 2612 raft_consensus.cc:695] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:02.556963 2612 consensus_queue.cc:237] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:02.558912 2172 catalog_manager.cc:5582] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:02.656775 2612 raft_consensus.cc:491] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:02.657038 2612 raft_consensus.cc:513] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.657766 2612 leader_election.cc:290] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:02.657929 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd1e744f16b407980acc76a77789727" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:02.658028 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd1e744f16b407980acc76a77789727" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:02.658170 2389 raft_consensus.cc:2466] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:02.658221 2463 raft_consensus.cc:2466] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:02.658564 2249 leader_election.cc:304] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:02.658836 2612 raft_consensus.cc:2802] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:02.658912 2612 raft_consensus.cc:491] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:02.658985 2612 raft_consensus.cc:3058] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.660636 2612 raft_consensus.cc:513] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.661185 2612 leader_election.cc:290] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:02.661608 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd1e744f16b407980acc76a77789727" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:02.661629 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd1e744f16b407980acc76a77789727" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:02.661867 2463 raft_consensus.cc:3058] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.661864 2389 raft_consensus.cc:3058] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:02.664505 2463 raft_consensus.cc:2466] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:02.664505 2389 raft_consensus.cc:2466] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:02.664963 2247 leader_election.cc:304] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:02.665180 2612 raft_consensus.cc:2802] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:02.665323 2612 raft_consensus.cc:695] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:02.665592 2612 consensus_queue.cc:237] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:02.667938 2172 catalog_manager.cc:5582] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:02.687616 2463 raft_consensus.cc:1273] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:02.687790 2464 raft_consensus.cc:1273] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:02.688246 2612 consensus_queue.cc:1035] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:02.688365 2389 raft_consensus.cc:1273] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:02.688292 2640 consensus_queue.cc:1035] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:02.688485 2388 raft_consensus.cc:1273] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:02.688930 2640 consensus_queue.cc:1035] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:02.689209 2612 consensus_queue.cc:1035] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:02.926165 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:03.178165 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 408ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:03.429652 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 661ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:03.682004 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 912ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:03.718705 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00087e+06us
I20250904 07:53:03.720000 2171 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:03.720754 2171 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:03.728092 2378 tablet_service.cc:1468] Processing CreateTablet for tablet f633b3dc86a6495e859a188cecc671d0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:03.728502 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f633b3dc86a6495e859a188cecc671d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.728415 2278 tablet_service.cc:1468] Processing CreateTablet for tablet f633b3dc86a6495e859a188cecc671d0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:03.728713 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 3d55b836537343439721fe076eff4ec0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:03.728793 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f633b3dc86a6495e859a188cecc671d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.729086 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d55b836537343439721fe076eff4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.729658 2453 tablet_service.cc:1468] Processing CreateTablet for tablet f633b3dc86a6495e859a188cecc671d0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:03.730028 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f633b3dc86a6495e859a188cecc671d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.730789 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 3d55b836537343439721fe076eff4ec0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:03.731153 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d55b836537343439721fe076eff4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.734033 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 3d55b836537343439721fe076eff4ec0 (DEFAULT_TABLE table=table.4 [id=a7a9378436454f2ebed9d044ff8bc7f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:03.734390 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d55b836537343439721fe076eff4ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:03.746717 2660 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:03.749104 2660 tablet_bootstrap.cc:654] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.751252 2659 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:03.753741 2659 tablet_bootstrap.cc:654] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.757889 2660 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:03.758008 2660 ts_tablet_manager.cc:1397] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250904 07:53:03.758603 2660 raft_consensus.cc:357] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.758771 2660 raft_consensus.cc:383] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.758854 2660 raft_consensus.cc:738] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.759057 2660 consensus_queue.cc:260] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.759501 2660 ts_tablet_manager.cc:1428] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:03.759860 2660 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:03.761812 2660 tablet_bootstrap.cc:654] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.763903 2661 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:03.764622 2659 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:03.764659 2660 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:03.764736 2660 ts_tablet_manager.cc:1397] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250904 07:53:03.764736 2659 ts_tablet_manager.cc:1397] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20250904 07:53:03.765373 2661 tablet_bootstrap.cc:654] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.765439 2660 raft_consensus.cc:357] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.765446 2659 raft_consensus.cc:357] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.765609 2660 raft_consensus.cc:383] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.765669 2660 raft_consensus.cc:738] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.765619 2659 raft_consensus.cc:383] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.765769 2659 raft_consensus.cc:738] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.765872 2660 consensus_queue.cc:260] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.765903 2659 consensus_queue.cc:260] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.766359 2660 ts_tablet_manager.cc:1428] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:03.766458 2659 ts_tablet_manager.cc:1428] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:03.766803 2659 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:03.768249 2659 tablet_bootstrap.cc:654] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.768285 2661 tablet_bootstrap.cc:492] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:03.768378 2661 ts_tablet_manager.cc:1397] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:53:03.769078 2661 raft_consensus.cc:357] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.769198 2661 raft_consensus.cc:383] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.769245 2661 raft_consensus.cc:738] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.769630 2661 consensus_queue.cc:260] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.769872 2659 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:03.770001 2659 ts_tablet_manager.cc:1397] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:03.770546 2659 raft_consensus.cc:357] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.770715 2659 raft_consensus.cc:383] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.770774 2659 raft_consensus.cc:738] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.771083 2659 consensus_queue.cc:260] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.771770 2659 ts_tablet_manager.cc:1428] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:03.773068 2639 raft_consensus.cc:491] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:03.773186 2639 raft_consensus.cc:513] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.773295 2661 ts_tablet_manager.cc:1428] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250904 07:53:03.773765 2639 leader_election.cc:290] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:03.773994 2661 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:03.774250 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d55b836537343439721fe076eff4ec0" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:03.774401 2464 raft_consensus.cc:2466] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:03.774497 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d55b836537343439721fe076eff4ec0" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:03.774669 2389 raft_consensus.cc:2466] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:03.774816 2247 leader_election.cc:304] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:03.775377 2661 tablet_bootstrap.cc:654] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:03.776379 2639 raft_consensus.cc:2802] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:03.776484 2639 raft_consensus.cc:491] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:03.776563 2639 raft_consensus.cc:3058] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.776902 2661 tablet_bootstrap.cc:492] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:03.777009 2661 ts_tablet_manager.cc:1397] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:03.777901 2661 raft_consensus.cc:357] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.778044 2661 raft_consensus.cc:383] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:03.778095 2661 raft_consensus.cc:738] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:03.778280 2661 consensus_queue.cc:260] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.778651 2661 ts_tablet_manager.cc:1428] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:03.778916 2639 raft_consensus.cc:513] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.779544 2639 leader_election.cc:290] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:03.779718 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d55b836537343439721fe076eff4ec0" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:03.779798 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d55b836537343439721fe076eff4ec0" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:03.779927 2389 raft_consensus.cc:3058] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.779935 2464 raft_consensus.cc:3058] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.782351 2389 raft_consensus.cc:2466] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:03.782356 2464 raft_consensus.cc:2466] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:03.782765 2247 leader_election.cc:304] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:03.783035 2639 raft_consensus.cc:2802] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:03.783370 2639 raft_consensus.cc:695] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:03.783640 2639 consensus_queue.cc:237] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:03.785939 2169 catalog_manager.cc:5582] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:03.788691 2617 raft_consensus.cc:491] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:03.788808 2617 raft_consensus.cc:513] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.789253 2617 leader_election.cc:290] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:03.789531 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f633b3dc86a6495e859a188cecc671d0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:03.789619 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f633b3dc86a6495e859a188cecc671d0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:03.789700 2464 raft_consensus.cc:2466] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:03.789819 2291 raft_consensus.cc:2466] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:03.790063 2345 leader_election.cc:304] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:03.790262 2617 raft_consensus.cc:2802] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:03.790356 2617 raft_consensus.cc:491] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:03.790438 2617 raft_consensus.cc:3058] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.792038 2617 raft_consensus.cc:513] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.792474 2617 leader_election.cc:290] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:03.792922 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f633b3dc86a6495e859a188cecc671d0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:03.793125 2291 raft_consensus.cc:3058] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.792992 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f633b3dc86a6495e859a188cecc671d0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:03.793598 2464 raft_consensus.cc:3058] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:03.794957 2291 raft_consensus.cc:2466] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:03.795328 2345 leader_election.cc:304] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:03.795370 2464 raft_consensus.cc:2466] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:03.795751 2617 raft_consensus.cc:2802] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:03.796169 2617 raft_consensus.cc:695] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:03.796443 2617 consensus_queue.cc:237] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:03.798708 2171 catalog_manager.cc:5582] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:03.810302 2464 raft_consensus.cc:1273] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:03.810334 2389 raft_consensus.cc:1273] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:03.810796 2669 consensus_queue.cc:1035] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:03.810981 2639 consensus_queue.cc:1035] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:03.811239 2291 raft_consensus.cc:1273] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:03.811570 2464 raft_consensus.cc:1273] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:03.812155 2617 consensus_queue.cc:1035] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:03.814224 2671 consensus_queue.cc:1035] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:03.818411 2674 mvcc.cc:204] Tried to move back new op lower bound from 7196558884086353920 to 7196558884032696320. Current Snapshot: MvccSnapshot[applied={T|T < 7196558884086353920}]
W20250904 07:53:04.134591 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 261ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:04.386638 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 514ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:04.639178 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 766ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:04.821763 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00071e+06us
I20250904 07:53:04.822604 2171 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:04.823284 2171 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:04.831082 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 3035b3b6d35d47d19186e2b55eb51012 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:04.831537 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3035b3b6d35d47d19186e2b55eb51012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.832008 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 3035b3b6d35d47d19186e2b55eb51012 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:04.832036 2378 tablet_service.cc:1468] Processing CreateTablet for tablet f6efeb21885f46108d1c5a44471095b4 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:04.832404 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3035b3b6d35d47d19186e2b55eb51012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.832502 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6efeb21885f46108d1c5a44471095b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.832577 2278 tablet_service.cc:1468] Processing CreateTablet for tablet f6efeb21885f46108d1c5a44471095b4 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:04.832960 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6efeb21885f46108d1c5a44471095b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.837249 2453 tablet_service.cc:1468] Processing CreateTablet for tablet f6efeb21885f46108d1c5a44471095b4 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:04.837684 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6efeb21885f46108d1c5a44471095b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.840099 2692 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:04.840636 2693 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:04.841764 2692 tablet_bootstrap.cc:654] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.841892 2693 tablet_bootstrap.cc:654] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.843806 2692 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:04.831431 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 3035b3b6d35d47d19186e2b55eb51012 (DEFAULT_TABLE table=table.5 [id=787cb9bd4dab487b81e1479f744e8d50]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:04.843982 2692 ts_tablet_manager.cc:1397] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:04.844214 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3035b3b6d35d47d19186e2b55eb51012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:04.845054 2692 raft_consensus.cc:357] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.846942 2692 raft_consensus.cc:383] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.847149 2692 raft_consensus.cc:738] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.847431 2692 consensus_queue.cc:260] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.849571 2692 ts_tablet_manager.cc:1428] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250904 07:53:04.850003 2692 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:04.851109 2696 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:04.852128 2693 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:04.852303 2693 ts_tablet_manager.cc:1397] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250904 07:53:04.853061 2692 tablet_bootstrap.cc:654] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.853204 2696 tablet_bootstrap.cc:654] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.853204 2693 raft_consensus.cc:357] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.853446 2693 raft_consensus.cc:383] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.853557 2693 raft_consensus.cc:738] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.853785 2693 consensus_queue.cc:260] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.860105 2693 ts_tablet_manager.cc:1428] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250904 07:53:04.861167 2693 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:04.861657 2696 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:04.861876 2696 ts_tablet_manager.cc:1397] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20250904 07:53:04.862661 2696 raft_consensus.cc:357] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.862906 2696 raft_consensus.cc:383] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.863052 2696 raft_consensus.cc:738] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.863456 2696 consensus_queue.cc:260] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.864403 2696 ts_tablet_manager.cc:1428] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:04.864704 2696 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:04.865415 2693 tablet_bootstrap.cc:654] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.866163 2696 tablet_bootstrap.cc:654] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:04.867761 2692 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:04.867931 2692 ts_tablet_manager.cc:1397] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.006s
I20250904 07:53:04.868084 2693 tablet_bootstrap.cc:492] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:04.868279 2693 ts_tablet_manager.cc:1397] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250904 07:53:04.868539 2692 raft_consensus.cc:357] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.868755 2692 raft_consensus.cc:383] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.869026 2692 raft_consensus.cc:738] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.869069 2693 raft_consensus.cc:357] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.869266 2693 raft_consensus.cc:383] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.869355 2693 raft_consensus.cc:738] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.869295 2692 consensus_queue.cc:260] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.869593 2693 consensus_queue.cc:260] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.869839 2692 ts_tablet_manager.cc:1428] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:04.870128 2693 ts_tablet_manager.cc:1428] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:04.870543 2696 tablet_bootstrap.cc:492] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:04.870730 2696 ts_tablet_manager.cc:1397] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250904 07:53:04.871346 2696 raft_consensus.cc:357] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.871968 2696 raft_consensus.cc:383] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:04.872109 2696 raft_consensus.cc:738] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:04.872375 2696 consensus_queue.cc:260] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:04.872939 2696 ts_tablet_manager.cc:1428] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:04.917923 2700 raft_consensus.cc:491] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:04.918112 2700 raft_consensus.cc:513] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.919018 2700 leader_election.cc:290] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:04.939121 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6efeb21885f46108d1c5a44471095b4" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:04.939375 2388 raft_consensus.cc:2466] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:04.939896 2427 leader_election.cc:304] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:04.939990 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6efeb21885f46108d1c5a44471095b4" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:04.940181 2291 raft_consensus.cc:2466] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:04.940179 2700 raft_consensus.cc:2802] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:04.940565 2700 raft_consensus.cc:491] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:04.940670 2700 raft_consensus.cc:3058] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:04.966634 2700 raft_consensus.cc:513] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.967269 2700 leader_election.cc:290] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:04.967626 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6efeb21885f46108d1c5a44471095b4" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:04.967816 2291 raft_consensus.cc:3058] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:04.968034 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6efeb21885f46108d1c5a44471095b4" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:04.968211 2388 raft_consensus.cc:3058] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:04.970070 2291 raft_consensus.cc:2466] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:04.970293 2388 raft_consensus.cc:2466] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:04.970409 2425 leader_election.cc:304] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:04.970679 2700 raft_consensus.cc:2802] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:04.971032 2700 raft_consensus.cc:695] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:04.971305 2700 consensus_queue.cc:237] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:04.973831 2169 catalog_manager.cc:5582] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:05.146344 2671 raft_consensus.cc:491] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:05.146590 2671 raft_consensus.cc:513] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:05.147147 2671 leader_election.cc:290] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:05.147521 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3035b3b6d35d47d19186e2b55eb51012" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:05.147548 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3035b3b6d35d47d19186e2b55eb51012" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:05.147727 2464 raft_consensus.cc:2466] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:05.147758 2291 raft_consensus.cc:2466] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:05.148092 2345 leader_election.cc:304] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:05.148340 2671 raft_consensus.cc:2802] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:05.148427 2671 raft_consensus.cc:491] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:05.148561 2671 raft_consensus.cc:3058] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:05.150650 2671 raft_consensus.cc:513] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:05.151154 2671 leader_election.cc:290] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:05.151470 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3035b3b6d35d47d19186e2b55eb51012" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:05.151495 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3035b3b6d35d47d19186e2b55eb51012" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:05.151661 2464 raft_consensus.cc:3058] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:05.151696 2291 raft_consensus.cc:3058] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:05.153579 2464 raft_consensus.cc:2466] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:05.153638 2291 raft_consensus.cc:2466] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:05.153918 2345 leader_election.cc:304] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:05.154142 2671 raft_consensus.cc:2802] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:05.154546 2671 raft_consensus.cc:695] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:05.154827 2671 consensus_queue.cc:237] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:05.157347 2169 catalog_manager.cc:5582] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:05.213390 2291 raft_consensus.cc:1273] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:05.214623 2709 consensus_queue.cc:1035] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:05.214736 2464 raft_consensus.cc:1273] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:05.215883 2709 consensus_queue.cc:1035] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:05.227176 2291 raft_consensus.cc:1273] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:05.227665 2700 consensus_queue.cc:1035] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:05.228799 2388 raft_consensus.cc:1273] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:05.229688 2700 consensus_queue.cc:1035] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:05.491101 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 215ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:05.745954 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 465ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:05.997558 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 719ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:06.227506 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00092e+06us
I20250904 07:53:06.228397 2171 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:06.229461 2171 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:06.240072 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 821862116ac24420a0b4103bca1968e7 (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:06.240527 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821862116ac24420a0b4103bca1968e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.240607 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 821862116ac24420a0b4103bca1968e7 (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:06.240819 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 8e123b0d29484c93abb544655194c12f (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:06.240972 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821862116ac24420a0b4103bca1968e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.241132 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e123b0d29484c93abb544655194c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.242973 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 821862116ac24420a0b4103bca1968e7 (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:06.243319 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821862116ac24420a0b4103bca1968e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.245615 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 8e123b0d29484c93abb544655194c12f (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:06.245961 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e123b0d29484c93abb544655194c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.248440 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 8e123b0d29484c93abb544655194c12f (DEFAULT_TABLE table=table.6 [id=94aee345356841dfafe0d5a3a082001f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:06.248853 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e123b0d29484c93abb544655194c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:06.261387 2731 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:06.263406 2731 tablet_bootstrap.cc:654] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.264789 2732 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:06.267489 2731 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:06.267658 2731 ts_tablet_manager.cc:1397] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250904 07:53:06.268350 2732 tablet_bootstrap.cc:654] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.268705 2731 raft_consensus.cc:357] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.268987 2731 raft_consensus.cc:383] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.269050 2731 raft_consensus.cc:738] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.269696 2731 consensus_queue.cc:260] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.270620 2731 ts_tablet_manager.cc:1428] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:06.271080 2731 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:06.273083 2731 tablet_bootstrap.cc:654] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.273103 2733 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:06.275056 2733 tablet_bootstrap.cc:654] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.277278 2733 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:06.277395 2733 ts_tablet_manager.cc:1397] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250904 07:53:06.277427 2731 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:06.277546 2731 ts_tablet_manager.cc:1397] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250904 07:53:06.278239 2732 tablet_bootstrap.cc:492] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:06.278255 2733 raft_consensus.cc:357] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.278582 2731 raft_consensus.cc:357] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.278522 2733 raft_consensus.cc:383] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.278496 2732 ts_tablet_manager.cc:1397] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20250904 07:53:06.278769 2733 raft_consensus.cc:738] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.279646 2733 consensus_queue.cc:260] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.279740 2732 raft_consensus.cc:357] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.279866 2732 raft_consensus.cc:383] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.279917 2732 raft_consensus.cc:738] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.278762 2731 raft_consensus.cc:383] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.280205 2731 raft_consensus.cc:738] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.280107 2732 consensus_queue.cc:260] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.280575 2733 ts_tablet_manager.cc:1428] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:06.280411 2731 consensus_queue.cc:260] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.280889 2732 ts_tablet_manager.cc:1428] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:06.281023 2731 ts_tablet_manager.cc:1428] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:06.281174 2732 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:06.281611 2733 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:06.283576 2733 tablet_bootstrap.cc:654] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.287974 2733 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:06.288130 2733 ts_tablet_manager.cc:1397] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250904 07:53:06.291100 2732 tablet_bootstrap.cc:654] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:06.289814 2733 raft_consensus.cc:357] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.291363 2733 raft_consensus.cc:383] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.291435 2733 raft_consensus.cc:738] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.291656 2733 consensus_queue.cc:260] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.293282 2733 ts_tablet_manager.cc:1428] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:53:06.295212 2732 tablet_bootstrap.cc:492] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:06.295361 2732 ts_tablet_manager.cc:1397] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250904 07:53:06.296056 2732 raft_consensus.cc:357] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.296268 2732 raft_consensus.cc:383] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:06.296339 2732 raft_consensus.cc:738] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:06.296552 2732 consensus_queue.cc:260] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.297065 2732 ts_tablet_manager.cc:1428] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:06.398017 2739 raft_consensus.cc:491] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:06.398151 2739 raft_consensus.cc:513] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.398628 2739 leader_election.cc:290] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:06.398924 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e123b0d29484c93abb544655194c12f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:06.398972 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e123b0d29484c93abb544655194c12f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:06.399132 2464 raft_consensus.cc:2466] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:06.399132 2388 raft_consensus.cc:2466] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:06.399574 2249 leader_election.cc:304] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:06.399834 2739 raft_consensus.cc:2802] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:06.399922 2739 raft_consensus.cc:491] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:06.399995 2739 raft_consensus.cc:3058] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.401847 2739 raft_consensus.cc:513] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.402302 2739 leader_election.cc:290] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:06.402555 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e123b0d29484c93abb544655194c12f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:06.402675 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e123b0d29484c93abb544655194c12f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:06.402716 2388 raft_consensus.cc:3058] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.402818 2464 raft_consensus.cc:3058] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.405396 2464 raft_consensus.cc:2466] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:06.405396 2388 raft_consensus.cc:2466] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:06.405738 2249 leader_election.cc:304] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:06.405997 2739 raft_consensus.cc:2802] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:06.406142 2739 raft_consensus.cc:695] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:06.406335 2739 consensus_queue.cc:237] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:06.408607 2171 catalog_manager.cc:5582] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:06.602922 2726 raft_consensus.cc:491] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:06.603065 2726 raft_consensus.cc:513] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.603623 2726 leader_election.cc:290] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:06.603924 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "821862116ac24420a0b4103bca1968e7" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:06.603994 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "821862116ac24420a0b4103bca1968e7" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:06.604075 2464 raft_consensus.cc:2466] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:06.604168 2291 raft_consensus.cc:2466] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:06.604432 2345 leader_election.cc:304] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:06.604681 2726 raft_consensus.cc:2802] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:06.604743 2726 raft_consensus.cc:491] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:06.604794 2726 raft_consensus.cc:3058] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.606587 2726 raft_consensus.cc:513] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.607064 2726 leader_election.cc:290] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:06.607342 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "821862116ac24420a0b4103bca1968e7" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:06.607338 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "821862116ac24420a0b4103bca1968e7" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:06.607484 2291 raft_consensus.cc:3058] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.607491 2464 raft_consensus.cc:3058] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:06.609982 2291 raft_consensus.cc:2466] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:06.610028 2464 raft_consensus.cc:2466] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:06.610306 2345 leader_election.cc:304] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:06.610508 2726 raft_consensus.cc:2802] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:06.610905 2726 raft_consensus.cc:695] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:06.611164 2726 consensus_queue.cc:237] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:06.613533 2172 catalog_manager.cc:5582] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:06.626526 2464 raft_consensus.cc:1273] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:06.626754 2291 raft_consensus.cc:1273] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:06.626874 2463 raft_consensus.cc:1273] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:06.627167 2698 consensus_queue.cc:1035] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:06.627319 2388 raft_consensus.cc:1273] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:06.627523 2726 consensus_queue.cc:1035] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:06.627722 2698 consensus_queue.cc:1035] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:06.628338 2726 consensus_queue.cc:1035] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:06.924600 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:07.177763 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 480ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:07.430065 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 733ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:07.646101 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00105e+06us
I20250904 07:53:07.647055 2169 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:07.647991 2169 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:07.660655 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa68c431d5c4e8a8d86d0295d024d5a (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:07.661164 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa68c431d5c4e8a8d86d0295d024d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.661998 2454 tablet_service.cc:1468] Processing CreateTablet for tablet af9f6b7c4b144844ba8aa9efe6f9d483 (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:07.662242 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa68c431d5c4e8a8d86d0295d024d5a (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:07.662251 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa68c431d5c4e8a8d86d0295d024d5a (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:07.662593 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa68c431d5c4e8a8d86d0295d024d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.662627 2280 tablet_service.cc:1468] Processing CreateTablet for tablet af9f6b7c4b144844ba8aa9efe6f9d483 (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:07.662885 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9f6b7c4b144844ba8aa9efe6f9d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.662343 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9f6b7c4b144844ba8aa9efe6f9d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.666033 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa68c431d5c4e8a8d86d0295d024d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.666172 2378 tablet_service.cc:1468] Processing CreateTablet for tablet af9f6b7c4b144844ba8aa9efe6f9d483 (DEFAULT_TABLE table=table.7 [id=0f3735c21022432b8c083e57211b4f1f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:07.668764 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9f6b7c4b144844ba8aa9efe6f9d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:07.678537 2765 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:07.682785 2765 tablet_bootstrap.cc:654] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.684238 2764 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:07.686079 2764 tablet_bootstrap.cc:654] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.688038 2766 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:07.688943 2765 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:07.689154 2765 ts_tablet_manager.cc:1397] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250904 07:53:07.689929 2766 tablet_bootstrap.cc:654] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.690006 2765 raft_consensus.cc:357] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.690454 2765 raft_consensus.cc:383] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.690587 2765 raft_consensus.cc:738] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.690883 2765 consensus_queue.cc:260] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.691547 2765 ts_tablet_manager.cc:1428] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:07.693158 2764 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:07.694845 2766 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:07.695032 2766 ts_tablet_manager.cc:1397] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250904 07:53:07.695775 2766 raft_consensus.cc:357] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.696015 2766 raft_consensus.cc:383] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.696115 2766 raft_consensus.cc:738] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.696393 2766 consensus_queue.cc:260] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.697196 2766 ts_tablet_manager.cc:1428] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:07.699137 2766 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:07.700423 2766 tablet_bootstrap.cc:654] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.700896 2764 ts_tablet_manager.cc:1397] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20250904 07:53:07.701617 2764 raft_consensus.cc:357] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.701864 2764 raft_consensus.cc:383] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.701970 2764 raft_consensus.cc:738] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.702143 2764 consensus_queue.cc:260] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.702474 2765 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:07.702603 2764 ts_tablet_manager.cc:1428] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:07.705228 2764 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:07.708130 2764 tablet_bootstrap.cc:654] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.709180 2766 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:07.709388 2766 ts_tablet_manager.cc:1397] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250904 07:53:07.710247 2766 raft_consensus.cc:357] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.710541 2766 raft_consensus.cc:383] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.710660 2766 raft_consensus.cc:738] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.710984 2766 consensus_queue.cc:260] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.711995 2765 tablet_bootstrap.cc:654] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:07.712754 2766 ts_tablet_manager.cc:1428] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:07.713776 2764 tablet_bootstrap.cc:492] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:07.713883 2764 ts_tablet_manager.cc:1397] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250904 07:53:07.714526 2764 raft_consensus.cc:357] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.714722 2764 raft_consensus.cc:383] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.714833 2764 raft_consensus.cc:738] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.715090 2764 consensus_queue.cc:260] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.715549 2764 ts_tablet_manager.cc:1428] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:07.723237 2765 tablet_bootstrap.cc:492] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:07.723343 2765 ts_tablet_manager.cc:1397] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20250904 07:53:07.723789 2765 raft_consensus.cc:357] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.723902 2765 raft_consensus.cc:383] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:07.723954 2765 raft_consensus.cc:738] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:07.724102 2765 consensus_queue.cc:260] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.724509 2765 ts_tablet_manager.cc:1428] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:07.831122 2698 raft_consensus.cc:491] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:07.831264 2698 raft_consensus.cc:513] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.831496 2726 raft_consensus.cc:491] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:07.831643 2726 raft_consensus.cc:513] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.831823 2698 leader_election.cc:290] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:07.832273 2726 leader_election.cc:290] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:07.832234 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9f6b7c4b144844ba8aa9efe6f9d483" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:07.832324 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9f6b7c4b144844ba8aa9efe6f9d483" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:07.832432 2389 raft_consensus.cc:2466] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:07.832458 2464 raft_consensus.cc:2466] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:07.832897 2249 leader_election.cc:304] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:07.832998 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa68c431d5c4e8a8d86d0295d024d5a" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:07.833127 2291 raft_consensus.cc:2466] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:07.833161 2698 raft_consensus.cc:2802] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:07.833230 2698 raft_consensus.cc:491] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:07.833312 2698 raft_consensus.cc:3058] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.833441 2345 leader_election.cc:304] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:07.833580 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa68c431d5c4e8a8d86d0295d024d5a" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:07.833664 2726 raft_consensus.cc:2802] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:07.833709 2464 raft_consensus.cc:2466] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:07.833750 2726 raft_consensus.cc:491] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:07.833808 2726 raft_consensus.cc:3058] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.835811 2698 raft_consensus.cc:513] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.835947 2726 raft_consensus.cc:513] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.836529 2726 leader_election.cc:290] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:07.836867 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9f6b7c4b144844ba8aa9efe6f9d483" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:07.836549 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9f6b7c4b144844ba8aa9efe6f9d483" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:07.836902 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa68c431d5c4e8a8d86d0295d024d5a" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:07.836984 2464 raft_consensus.cc:3058] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.836902 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa68c431d5c4e8a8d86d0295d024d5a" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:07.837045 2463 raft_consensus.cc:3058] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.837121 2291 raft_consensus.cc:3058] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.837044 2389 raft_consensus.cc:3058] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:07.839354 2389 raft_consensus.cc:2466] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:07.839659 2249 leader_election.cc:304] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:07.839704 2464 raft_consensus.cc:2466] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:07.839704 2291 raft_consensus.cc:2466] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:07.839929 2761 raft_consensus.cc:2802] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:07.840152 2345 leader_election.cc:304] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:07.840412 2761 raft_consensus.cc:695] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:07.840744 2761 consensus_queue.cc:237] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:07.836791 2698 leader_election.cc:290] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:07.841246 2463 raft_consensus.cc:2466] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:07.840401 2726 raft_consensus.cc:2802] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:07.844434 2171 catalog_manager.cc:5582] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:07.849057 2726 raft_consensus.cc:695] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:07.849390 2726 consensus_queue.cc:237] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:07.852306 2171 catalog_manager.cc:5582] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:07.872519 2389 raft_consensus.cc:1273] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:07.872951 2463 raft_consensus.cc:1273] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:07.872961 2698 consensus_queue.cc:1035] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:07.873401 2698 consensus_queue.cc:1035] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:07.874127 2464 raft_consensus.cc:1273] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:07.874130 2291 raft_consensus.cc:1273] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:07.874575 2726 consensus_queue.cc:1035] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:07.874716 2776 consensus_queue.cc:1035] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:07.883672 2778 mvcc.cc:204] Tried to move back new op lower bound from 7196558900727508992 to 7196558900634173440. Current Snapshot: MvccSnapshot[applied={T|T < 7196558900727508992}]
W20250904 07:53:08.173935 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 238ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:08.428609 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 490ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:08.681536 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 744ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:08.885423 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00094e+06us
I20250904 07:53:08.886348 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:08.887312 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:08.895907 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:08.896296 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.896425 2278 tablet_service.cc:1468] Processing CreateTablet for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0 (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:08.896528 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:08.896759 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.897053 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.898483 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:08.899060 2453 tablet_service.cc:1468] Processing CreateTablet for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0 (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:08.899423 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.899838 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1766b3b4704b1fb8d60acf06c9c7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.900355 2379 tablet_service.cc:1468] Processing CreateTablet for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0 (DEFAULT_TABLE table=table.8 [id=9d583599857643289a28a90bafb7cab2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:08.900779 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c016ef01d8eb4df4a8cf9fbcbf7ba1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:08.910195 2796 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:08.911976 2796 tablet_bootstrap.cc:654] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.925268 2799 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:08.925796 2798 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:08.927448 2799 tablet_bootstrap.cc:654] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.931224 2798 tablet_bootstrap.cc:654] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.932720 2796 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:08.932838 2796 ts_tablet_manager.cc:1397] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20250904 07:53:08.934628 2796 raft_consensus.cc:357] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.934813 2796 raft_consensus.cc:383] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.934876 2796 raft_consensus.cc:738] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.935042 2796 consensus_queue.cc:260] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.936817 2799 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:08.936969 2799 ts_tablet_manager.cc:1397] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20250904 07:53:08.942544 2799 raft_consensus.cc:357] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.942761 2799 raft_consensus.cc:383] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.942843 2799 raft_consensus.cc:738] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.943071 2799 consensus_queue.cc:260] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.943795 2799 ts_tablet_manager.cc:1428] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250904 07:53:08.944576 2796 ts_tablet_manager.cc:1428] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250904 07:53:08.944905 2796 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:08.946185 2725 raft_consensus.cc:491] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:08.946358 2725 raft_consensus.cc:513] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.946559 2796 tablet_bootstrap.cc:654] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.947042 2725 leader_election.cc:290] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:08.944643 2799 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:08.947882 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
W20250904 07:53:08.948663 2427 leader_election.cc:343] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Illegal state: must be running to vote when last-logged opid is not known
I20250904 07:53:08.949234 2799 tablet_bootstrap.cc:654] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.951298 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
W20250904 07:53:08.951934 2425 leader_election.cc:343] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Illegal state: must be running to vote when last-logged opid is not known
I20250904 07:53:08.952049 2425 leader_election.cc:304] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837; no voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81
I20250904 07:53:08.952529 2725 raft_consensus.cc:2747] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:53:08.954228 2798 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:08.954495 2798 ts_tablet_manager.cc:1397] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250904 07:53:08.955149 2796 tablet_bootstrap.cc:492] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:08.955293 2796 ts_tablet_manager.cc:1397] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250904 07:53:08.955503 2798 raft_consensus.cc:357] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.955765 2798 raft_consensus.cc:383] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.955878 2798 raft_consensus.cc:738] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.956035 2796 raft_consensus.cc:357] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.956223 2796 raft_consensus.cc:383] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.956264 2798 consensus_queue.cc:260] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.956287 2796 raft_consensus.cc:738] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.956732 2796 consensus_queue.cc:260] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:08.957234 2799 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:08.957391 2799 ts_tablet_manager.cc:1397] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20250904 07:53:08.957231 2796 ts_tablet_manager.cc:1428] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:08.957942 2798 ts_tablet_manager.cc:1428] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:08.958076 2799 raft_consensus.cc:357] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.958158 2798 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:08.959923 2798 tablet_bootstrap.cc:654] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:08.958201 2799 raft_consensus.cc:383] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.960055 2799 raft_consensus.cc:738] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.960264 2799 consensus_queue.cc:260] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.963150 2799 ts_tablet_manager.cc:1428] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250904 07:53:08.965703 2798 tablet_bootstrap.cc:492] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:08.965833 2798 ts_tablet_manager.cc:1397] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20250904 07:53:08.966315 2798 raft_consensus.cc:357] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.966429 2798 raft_consensus.cc:383] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:08.966522 2798 raft_consensus.cc:738] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:08.966789 2798 consensus_queue.cc:260] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:08.967378 2798 ts_tablet_manager.cc:1428] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:09.043298 2725 raft_consensus.cc:491] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:09.043435 2725 raft_consensus.cc:513] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:09.044027 2725 leader_election.cc:290] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:09.044366 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d1766b3b4704b1fb8d60acf06c9c7eb" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:09.044404 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d1766b3b4704b1fb8d60acf06c9c7eb" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:09.044540 2389 raft_consensus.cc:2466] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:09.044601 2291 raft_consensus.cc:2466] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:09.044911 2427 leader_election.cc:304] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:09.045146 2725 raft_consensus.cc:2802] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:09.045264 2725 raft_consensus.cc:491] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:09.045331 2725 raft_consensus.cc:3058] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.047441 2725 raft_consensus.cc:513] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:09.047876 2725 leader_election.cc:290] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:09.048213 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d1766b3b4704b1fb8d60acf06c9c7eb" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:09.048256 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d1766b3b4704b1fb8d60acf06c9c7eb" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:09.048365 2389 raft_consensus.cc:3058] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.048446 2291 raft_consensus.cc:3058] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.050223 2291 raft_consensus.cc:2466] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:09.050266 2389 raft_consensus.cc:2466] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:09.050547 2427 leader_election.cc:304] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:09.050817 2725 raft_consensus.cc:2802] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:09.051692 2725 raft_consensus.cc:695] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:09.051988 2725 consensus_queue.cc:237] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:09.054668 2168 catalog_manager.cc:5582] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:09.290350 2802 raft_consensus.cc:491] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:09.290529 2802 raft_consensus.cc:513] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:09.291308 2802 leader_election.cc:290] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:09.291700 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:09.292119 2463 raft_consensus.cc:2466] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:09.292294 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:09.292512 2291 raft_consensus.cc:2466] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:09.292591 2345 leader_election.cc:304] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:09.293104 2802 raft_consensus.cc:2802] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:09.293226 2802 raft_consensus.cc:491] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:09.293323 2802 raft_consensus.cc:3058] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.295773 2802 raft_consensus.cc:513] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:09.296396 2802 leader_election.cc:290] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:09.296636 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:09.296770 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c016ef01d8eb4df4a8cf9fbcbf7ba1e0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:09.296816 2291 raft_consensus.cc:3058] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.296985 2463 raft_consensus.cc:3058] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:09.298801 2291 raft_consensus.cc:2466] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:09.298802 2463 raft_consensus.cc:2466] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:09.299171 2345 leader_election.cc:304] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:09.299432 2802 raft_consensus.cc:2802] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:09.299588 2802 raft_consensus.cc:695] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:09.299849 2802 consensus_queue.cc:237] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:09.302191 2169 catalog_manager.cc:5582] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:09.359644 2389 raft_consensus.cc:1273] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:09.359936 2291 raft_consensus.cc:1273] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:09.359989 2463 raft_consensus.cc:1273] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:09.360136 2292 raft_consensus.cc:1273] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:09.360527 2808 consensus_queue.cc:1035] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:09.360582 2803 consensus_queue.cc:1035] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:09.360751 2802 consensus_queue.cc:1035] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:09.361160 2725 consensus_queue.cc:1035] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:09.631582 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 204ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:09.881713 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 457ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:10.135352 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 707ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:10.375402 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.0009e+06us
I20250904 07:53:10.376246 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:10.377214 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:10.386660 2378 tablet_service.cc:1468] Processing CreateTablet for tablet f4603a3c045a4016b557456a09da845c (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:10.387027 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4603a3c045a4016b557456a09da845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.389300 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3 (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:10.389752 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.392390 2280 tablet_service.cc:1468] Processing CreateTablet for tablet f4603a3c045a4016b557456a09da845c (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:10.392504 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3 (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:10.392736 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4603a3c045a4016b557456a09da845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.393349 2829 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:10.393780 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3 (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:10.394127 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.394894 2829 tablet_bootstrap.cc:654] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.394923 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa8abb8eb304fe7be4ade8a1afbd9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.397163 2454 tablet_service.cc:1468] Processing CreateTablet for tablet f4603a3c045a4016b557456a09da845c (DEFAULT_TABLE table=table.9 [id=20b00f9f85fe495885b79ff5e19a0ec2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:10.397509 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4603a3c045a4016b557456a09da845c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:10.407473 2831 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:10.410691 2831 tablet_bootstrap.cc:654] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.412880 2833 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:10.413136 2829 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:10.413259 2829 ts_tablet_manager.cc:1397] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20250904 07:53:10.414037 2829 raft_consensus.cc:357] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.414386 2829 raft_consensus.cc:383] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.414469 2829 raft_consensus.cc:738] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.414710 2829 consensus_queue.cc:260] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.415165 2831 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:10.415241 2829 ts_tablet_manager.cc:1428] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:10.415341 2831 ts_tablet_manager.cc:1397] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20250904 07:53:10.415527 2829 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:10.416088 2831 raft_consensus.cc:357] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.416308 2831 raft_consensus.cc:383] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.416327 2833 tablet_bootstrap.cc:654] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.416451 2831 raft_consensus.cc:738] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.417647 2831 consensus_queue.cc:260] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.418287 2829 tablet_bootstrap.cc:654] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.418531 2831 ts_tablet_manager.cc:1428] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:10.418804 2831 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:10.420189 2831 tablet_bootstrap.cc:654] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.427716 2831 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:10.427835 2831 ts_tablet_manager.cc:1397] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20250904 07:53:10.427991 2829 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:10.428524 2829 ts_tablet_manager.cc:1397] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20250904 07:53:10.428843 2831 raft_consensus.cc:357] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.429040 2831 raft_consensus.cc:383] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.429100 2831 raft_consensus.cc:738] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.429102 2829 raft_consensus.cc:357] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.429342 2829 raft_consensus.cc:383] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.429296 2831 consensus_queue.cc:260] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.429411 2829 raft_consensus.cc:738] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.429738 2833 tablet_bootstrap.cc:492] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:10.429646 2829 consensus_queue.cc:260] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.429890 2833 ts_tablet_manager.cc:1397] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20250904 07:53:10.430265 2829 ts_tablet_manager.cc:1428] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:10.430558 2833 raft_consensus.cc:357] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.430687 2833 raft_consensus.cc:383] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.429883 2831 ts_tablet_manager.cc:1428] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:10.430756 2833 raft_consensus.cc:738] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.431177 2833 consensus_queue.cc:260] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.432441 2833 ts_tablet_manager.cc:1428] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:10.433565 2833 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:10.436755 2833 tablet_bootstrap.cc:654] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:10.438221 2833 tablet_bootstrap.cc:492] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:10.438334 2833 ts_tablet_manager.cc:1397] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250904 07:53:10.438827 2833 raft_consensus.cc:357] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.438979 2833 raft_consensus.cc:383] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:10.439045 2833 raft_consensus.cc:738] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:10.439188 2833 consensus_queue.cc:260] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.439491 2833 ts_tablet_manager.cc:1428] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:10.489228 2803 raft_consensus.cc:491] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:10.489394 2803 raft_consensus.cc:513] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.489905 2803 leader_election.cc:290] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:10.490252 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4603a3c045a4016b557456a09da845c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:10.490276 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4603a3c045a4016b557456a09da845c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:10.490478 2292 raft_consensus.cc:2466] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:10.490478 2463 raft_consensus.cc:2466] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:10.490841 2345 leader_election.cc:304] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:10.491045 2803 raft_consensus.cc:2802] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:10.491122 2803 raft_consensus.cc:491] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:10.491200 2803 raft_consensus.cc:3058] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.493458 2803 raft_consensus.cc:513] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.494146 2803 leader_election.cc:290] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:10.494395 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4603a3c045a4016b557456a09da845c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:10.494541 2463 raft_consensus.cc:3058] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.494525 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4603a3c045a4016b557456a09da845c" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:10.494693 2292 raft_consensus.cc:3058] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.496526 2463 raft_consensus.cc:2466] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:10.496876 2345 leader_election.cc:304] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:10.497115 2803 raft_consensus.cc:2802] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:10.497366 2292 raft_consensus.cc:2466] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:10.497496 2803 raft_consensus.cc:695] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:10.497762 2803 consensus_queue.cc:237] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:10.500252 2168 catalog_manager.cc:5582] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:10.523571 2822 raft_consensus.cc:491] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:10.523726 2822 raft_consensus.cc:513] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.524389 2822 leader_election.cc:290] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:10.524624 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fa8abb8eb304fe7be4ade8a1afbd9e3" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:10.524787 2292 raft_consensus.cc:2466] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:10.524766 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fa8abb8eb304fe7be4ade8a1afbd9e3" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:10.524987 2389 raft_consensus.cc:2466] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:10.525209 2425 leader_election.cc:304] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:10.525413 2822 raft_consensus.cc:2802] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:10.525498 2822 raft_consensus.cc:491] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:10.525570 2822 raft_consensus.cc:3058] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.527853 2822 raft_consensus.cc:513] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.528381 2822 leader_election.cc:290] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:10.528625 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fa8abb8eb304fe7be4ade8a1afbd9e3" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:10.528780 2292 raft_consensus.cc:3058] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.528743 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fa8abb8eb304fe7be4ade8a1afbd9e3" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:10.528962 2389 raft_consensus.cc:3058] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:10.531353 2292 raft_consensus.cc:2466] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:10.531522 2389 raft_consensus.cc:2466] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:10.531679 2425 leader_election.cc:304] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:10.531965 2822 raft_consensus.cc:2802] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:10.532119 2822 raft_consensus.cc:695] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:10.532370 2822 consensus_queue.cc:237] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:10.535073 2170 catalog_manager.cc:5582] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:10.568480 2389 raft_consensus.cc:1273] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.568750 2292 raft_consensus.cc:1273] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.569062 2463 raft_consensus.cc:1273] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.569235 2822 consensus_queue.cc:1035] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:10.569411 2292 raft_consensus.cc:1273] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:10.569455 2841 consensus_queue.cc:1035] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:10.569469 2803 consensus_queue.cc:1035] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:10.569933 2803 consensus_queue.cc:1035] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:10.847707 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 204ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:11.104844 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 458ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:11.357924 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:11.590606 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.0009e+06us
I20250904 07:53:11.591415 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:11.592061 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:11.603681 2378 tablet_service.cc:1468] Processing CreateTablet for tablet df55f6a7bfd84f01960d43ca38a627c3 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:11.604099 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df55f6a7bfd84f01960d43ca38a627c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.604245 2379 tablet_service.cc:1468] Processing CreateTablet for tablet eaa1415f6508476db015383017a473b0 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:11.604668 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa1415f6508476db015383017a473b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.608553 2863 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:11.609182 2454 tablet_service.cc:1468] Processing CreateTablet for tablet df55f6a7bfd84f01960d43ca38a627c3 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:11.609499 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df55f6a7bfd84f01960d43ca38a627c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.609813 2278 tablet_service.cc:1468] Processing CreateTablet for tablet df55f6a7bfd84f01960d43ca38a627c3 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:11.609966 2863 tablet_bootstrap.cc:654] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.610191 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df55f6a7bfd84f01960d43ca38a627c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.613144 2453 tablet_service.cc:1468] Processing CreateTablet for tablet eaa1415f6508476db015383017a473b0 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:11.613539 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa1415f6508476db015383017a473b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.616205 2280 tablet_service.cc:1468] Processing CreateTablet for tablet eaa1415f6508476db015383017a473b0 (DEFAULT_TABLE table=table.10 [id=d223a1ed04644451b8bbd22109602394]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:11.616518 2863 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:11.616559 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa1415f6508476db015383017a473b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:11.616631 2863 ts_tablet_manager.cc:1397] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250904 07:53:11.617395 2863 raft_consensus.cc:357] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.617589 2863 raft_consensus.cc:383] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.617642 2863 raft_consensus.cc:738] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.617853 2863 consensus_queue.cc:260] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.618361 2863 ts_tablet_manager.cc:1428] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:11.619313 2863 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:11.620764 2863 tablet_bootstrap.cc:654] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.626060 2865 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:11.627766 2865 tablet_bootstrap.cc:654] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.632380 2865 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:11.632566 2865 ts_tablet_manager.cc:1397] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20250904 07:53:11.633447 2865 raft_consensus.cc:357] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.633693 2865 raft_consensus.cc:383] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.633800 2865 raft_consensus.cc:738] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.634037 2865 consensus_queue.cc:260] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.635388 2865 ts_tablet_manager.cc:1428] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:11.635695 2865 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:11.636962 2866 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:11.637786 2863 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:11.637989 2863 ts_tablet_manager.cc:1397] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20250904 07:53:11.638208 2865 tablet_bootstrap.cc:654] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.638540 2863 raft_consensus.cc:357] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.638676 2863 raft_consensus.cc:383] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.638724 2863 raft_consensus.cc:738] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.638959 2863 consensus_queue.cc:260] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.640327 2863 ts_tablet_manager.cc:1428] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:11.640578 2866 tablet_bootstrap.cc:654] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.644145 2865 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:11.644300 2865 ts_tablet_manager.cc:1397] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20250904 07:53:11.644562 2866 tablet_bootstrap.cc:492] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:11.644670 2866 ts_tablet_manager.cc:1397] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250904 07:53:11.645011 2865 raft_consensus.cc:357] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.645177 2865 raft_consensus.cc:383] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.645241 2865 raft_consensus.cc:738] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.645391 2866 raft_consensus.cc:357] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.645432 2865 consensus_queue.cc:260] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.645537 2866 raft_consensus.cc:383] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.645592 2866 raft_consensus.cc:738] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.645774 2866 consensus_queue.cc:260] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.645922 2865 ts_tablet_manager.cc:1428] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:11.646206 2866 ts_tablet_manager.cc:1428] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:11.646530 2866 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:11.647989 2866 tablet_bootstrap.cc:654] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:11.649461 2866 tablet_bootstrap.cc:492] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:11.649643 2866 ts_tablet_manager.cc:1397] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20250904 07:53:11.650290 2866 raft_consensus.cc:357] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.650493 2866 raft_consensus.cc:383] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:11.650600 2866 raft_consensus.cc:738] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:11.650816 2866 consensus_queue.cc:260] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.651367 2866 ts_tablet_manager.cc:1428] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:11.839776 2858 raft_consensus.cc:491] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:11.839901 2858 raft_consensus.cc:513] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.840332 2858 leader_election.cc:290] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:11.840668 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df55f6a7bfd84f01960d43ca38a627c3" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:11.840718 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df55f6a7bfd84f01960d43ca38a627c3" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:11.840871 2463 raft_consensus.cc:2466] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:11.840868 2291 raft_consensus.cc:2466] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:11.841240 2345 leader_election.cc:304] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:11.841413 2858 raft_consensus.cc:2802] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:11.841492 2858 raft_consensus.cc:491] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:11.841560 2858 raft_consensus.cc:3058] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.843132 2858 raft_consensus.cc:513] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.843519 2858 leader_election.cc:290] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:11.843912 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df55f6a7bfd84f01960d43ca38a627c3" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:11.843942 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df55f6a7bfd84f01960d43ca38a627c3" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:11.844066 2463 raft_consensus.cc:3058] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.844094 2291 raft_consensus.cc:3058] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.846467 2463 raft_consensus.cc:2466] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:11.846464 2291 raft_consensus.cc:2466] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:11.846762 2345 leader_election.cc:304] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:11.846953 2858 raft_consensus.cc:2802] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:11.847060 2858 raft_consensus.cc:695] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:11.847258 2858 consensus_queue.cc:237] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:11.849884 2858 raft_consensus.cc:491] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:11.850019 2858 raft_consensus.cc:513] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.850168 2169 catalog_manager.cc:5582] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:11.850718 2858 leader_election.cc:290] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:11.850766 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa1415f6508476db015383017a473b0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:11.850958 2291 raft_consensus.cc:2466] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:11.851048 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa1415f6508476db015383017a473b0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:11.851191 2463 raft_consensus.cc:2466] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:11.851287 2345 leader_election.cc:304] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:11.851537 2858 raft_consensus.cc:2802] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:11.851645 2858 raft_consensus.cc:491] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:11.851732 2858 raft_consensus.cc:3058] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.853484 2858 raft_consensus.cc:513] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.853938 2858 leader_election.cc:290] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:11.854188 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa1415f6508476db015383017a473b0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:11.854357 2291 raft_consensus.cc:3058] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.854354 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa1415f6508476db015383017a473b0" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:11.854554 2463 raft_consensus.cc:3058] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:11.856653 2291 raft_consensus.cc:2466] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:11.856957 2345 leader_election.cc:304] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:11.857141 2858 raft_consensus.cc:2802] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:11.857147 2463 raft_consensus.cc:2466] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:11.857331 2858 raft_consensus.cc:695] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:11.857586 2858 consensus_queue.cc:237] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:11.859985 2169 catalog_manager.cc:5582] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:11.918221 2463 raft_consensus.cc:1273] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:11.919422 2291 raft_consensus.cc:1273] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:11.919509 2292 raft_consensus.cc:1273] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:11.919837 2463 raft_consensus.cc:1273] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:11.920132 2858 consensus_queue.cc:1035] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:11.920461 2858 consensus_queue.cc:1035] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:11.920603 2853 consensus_queue.cc:1035] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:11.920809 2858 consensus_queue.cc:1035] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:12.268814 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 271ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:12.520362 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 523ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:12.772500 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 775ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:12.946395 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00067e+06us
I20250904 07:53:12.947516 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:12.948271 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:12.955534 2378 tablet_service.cc:1468] Processing CreateTablet for tablet afcd723aff4f478d96a0a8992d940916 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:12.955965 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcd723aff4f478d96a0a8992d940916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.957311 2379 tablet_service.cc:1468] Processing CreateTablet for tablet b78c6741bce34db99dc3db5bf7db5ad6 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:12.957566 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78c6741bce34db99dc3db5bf7db5ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.960731 2896 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:12.961159 2454 tablet_service.cc:1468] Processing CreateTablet for tablet afcd723aff4f478d96a0a8992d940916 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:12.961483 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcd723aff4f478d96a0a8992d940916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.962350 2896 tablet_bootstrap.cc:654] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.963670 2278 tablet_service.cc:1468] Processing CreateTablet for tablet afcd723aff4f478d96a0a8992d940916 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:12.963963 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcd723aff4f478d96a0a8992d940916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.964205 2280 tablet_service.cc:1468] Processing CreateTablet for tablet b78c6741bce34db99dc3db5bf7db5ad6 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:12.964493 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78c6741bce34db99dc3db5bf7db5ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.966097 2453 tablet_service.cc:1468] Processing CreateTablet for tablet b78c6741bce34db99dc3db5bf7db5ad6 (DEFAULT_TABLE table=table.11 [id=4ef73d5200624e25b76878aecffb6814]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:12.966424 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78c6741bce34db99dc3db5bf7db5ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:12.968168 2898 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:12.969810 2896 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:12.969969 2896 ts_tablet_manager.cc:1397] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250904 07:53:12.970647 2898 tablet_bootstrap.cc:654] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.970923 2896 raft_consensus.cc:357] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:12.971115 2896 raft_consensus.cc:383] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.971200 2896 raft_consensus.cc:738] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.971431 2896 consensus_queue.cc:260] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:12.972302 2896 ts_tablet_manager.cc:1428] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:12.972663 2896 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:12.974960 2896 tablet_bootstrap.cc:654] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.984826 2896 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:12.984998 2898 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:12.985024 2896 ts_tablet_manager.cc:1397] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20250904 07:53:12.985093 2898 ts_tablet_manager.cc:1397] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20250904 07:53:12.985867 2896 raft_consensus.cc:357] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.986119 2896 raft_consensus.cc:383] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.986119 2898 raft_consensus.cc:357] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:12.986294 2898 raft_consensus.cc:383] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.986346 2898 raft_consensus.cc:738] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.986241 2896 raft_consensus.cc:738] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.986642 2898 consensus_queue.cc:260] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:12.986708 2896 consensus_queue.cc:260] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.987265 2898 ts_tablet_manager.cc:1428] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:12.987514 2898 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:12.987262 2896 ts_tablet_manager.cc:1428] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:12.990921 2898 tablet_bootstrap.cc:654] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.993207 2900 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:12.994244 2900 tablet_bootstrap.cc:654] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:12.995918 2898 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:12.996002 2900 tablet_bootstrap.cc:492] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:12.996038 2898 ts_tablet_manager.cc:1397] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20250904 07:53:12.996114 2900 ts_tablet_manager.cc:1397] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:12.996490 2898 raft_consensus.cc:357] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.996598 2898 raft_consensus.cc:383] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.996627 2898 raft_consensus.cc:738] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.996574 2900 raft_consensus.cc:357] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.996677 2900 raft_consensus.cc:383] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:12.996716 2900 raft_consensus.cc:738] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:12.996740 2898 consensus_queue.cc:260] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.996958 2900 consensus_queue.cc:260] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:12.997109 2898 ts_tablet_manager.cc:1428] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:12.997462 2900 ts_tablet_manager.cc:1428] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:12.997771 2900 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:12.999148 2900 tablet_bootstrap.cc:654] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:13.001802 2900 tablet_bootstrap.cc:492] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:13.002002 2900 ts_tablet_manager.cc:1397] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250904 07:53:13.002698 2900 raft_consensus.cc:357] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:13.002869 2900 raft_consensus.cc:383] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:13.002947 2900 raft_consensus.cc:738] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:13.003161 2900 consensus_queue.cc:260] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:13.003613 2900 ts_tablet_manager.cc:1428] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:13.104244 2876 raft_consensus.cc:491] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:13.104377 2876 raft_consensus.cc:513] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:13.104846 2876 leader_election.cc:290] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:13.105160 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78c6741bce34db99dc3db5bf7db5ad6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:13.105204 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78c6741bce34db99dc3db5bf7db5ad6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:13.105340 2464 raft_consensus.cc:2466] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:13.105340 2292 raft_consensus.cc:2466] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:13.105700 2345 leader_election.cc:304] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:13.105916 2876 raft_consensus.cc:2802] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:13.105994 2876 raft_consensus.cc:491] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:13.106058 2876 raft_consensus.cc:3058] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.108392 2876 raft_consensus.cc:513] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:13.108840 2876 leader_election.cc:290] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:13.109172 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78c6741bce34db99dc3db5bf7db5ad6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:13.109212 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b78c6741bce34db99dc3db5bf7db5ad6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:13.109344 2292 raft_consensus.cc:3058] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.109344 2464 raft_consensus.cc:3058] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.112185 2464 raft_consensus.cc:2466] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:13.112185 2292 raft_consensus.cc:2466] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:13.112548 2345 leader_election.cc:304] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:13.112749 2876 raft_consensus.cc:2802] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:13.113178 2876 raft_consensus.cc:695] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:13.113535 2876 consensus_queue.cc:237] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:13.116572 2170 catalog_manager.cc:5582] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:13.278815 2907 raft_consensus.cc:491] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:13.278944 2907 raft_consensus.cc:513] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:13.279398 2907 leader_election.cc:290] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:13.279691 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afcd723aff4f478d96a0a8992d940916" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:13.279737 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afcd723aff4f478d96a0a8992d940916" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:13.279865 2463 raft_consensus.cc:2466] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:13.279863 2388 raft_consensus.cc:2466] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:13.280253 2247 leader_election.cc:304] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:13.280496 2907 raft_consensus.cc:2802] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:13.280583 2907 raft_consensus.cc:491] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:13.280679 2907 raft_consensus.cc:3058] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.282884 2907 raft_consensus.cc:513] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:13.283324 2907 leader_election.cc:290] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:13.283644 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afcd723aff4f478d96a0a8992d940916" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:13.283759 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afcd723aff4f478d96a0a8992d940916" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:13.283804 2388 raft_consensus.cc:3058] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.283926 2463 raft_consensus.cc:3058] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:13.286082 2463 raft_consensus.cc:2466] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:13.286227 2388 raft_consensus.cc:2466] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:13.286516 2247 leader_election.cc:304] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:13.286780 2907 raft_consensus.cc:2802] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:13.286973 2907 raft_consensus.cc:695] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:13.287245 2907 consensus_queue.cc:237] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:13.290341 2170 catalog_manager.cc:5582] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:13.338871 2463 raft_consensus.cc:1273] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:13.338941 2388 raft_consensus.cc:1273] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:13.339387 2878 consensus_queue.cc:1035] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:13.340227 2464 raft_consensus.cc:1273] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:13.340446 2292 raft_consensus.cc:1273] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:13.340914 2876 consensus_queue.cc:1035] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:13.341416 2878 consensus_queue.cc:1035] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:13.342080 2905 consensus_queue.cc:1035] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:13.346043 2911 mvcc.cc:204] Tried to move back new op lower bound from 7196558923111325696 to 7196558922907029504. Current Snapshot: MvccSnapshot[applied={T|T < 7196558923111325696}]
W20250904 07:53:13.630760 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 219ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:13.887017 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 476ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:14.140478 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 733ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:14.355276 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00087e+06us
I20250904 07:53:14.356163 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:14.356869 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:14.367326 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9c55548b294b4895483709aa0cd36d (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:14.367734 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9c55548b294b4895483709aa0cd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.368484 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 4911b93f2bd1424284b0517f51d64b65 (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:14.369014 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911b93f2bd1424284b0517f51d64b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.371789 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9c55548b294b4895483709aa0cd36d (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:14.372092 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9c55548b294b4895483709aa0cd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.373368 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9c55548b294b4895483709aa0cd36d (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:14.373493 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 4911b93f2bd1424284b0517f51d64b65 (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:14.373682 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9c55548b294b4895483709aa0cd36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.373816 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911b93f2bd1424284b0517f51d64b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.375613 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 4911b93f2bd1424284b0517f51d64b65 (DEFAULT_TABLE table=table.12 [id=58122b9b10a84b0a980db12131f9a027]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:14.375924 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911b93f2bd1424284b0517f51d64b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:14.392351 2929 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:14.394846 2929 tablet_bootstrap.cc:654] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.397609 2928 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:14.398557 2928 tablet_bootstrap.cc:654] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.399288 2927 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:14.401157 2927 tablet_bootstrap.cc:654] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.403333 2928 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:14.403488 2928 ts_tablet_manager.cc:1397] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250904 07:53:14.409348 2928 raft_consensus.cc:357] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.409529 2928 raft_consensus.cc:383] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.409592 2928 raft_consensus.cc:738] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.409919 2927 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:14.410038 2927 ts_tablet_manager.cc:1397] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250904 07:53:14.410687 2927 raft_consensus.cc:357] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.410862 2927 raft_consensus.cc:383] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.410926 2927 raft_consensus.cc:738] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.410952 2928 consensus_queue.cc:260] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.411123 2927 consensus_queue.cc:260] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.411564 2928 ts_tablet_manager.cc:1428] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250904 07:53:14.411926 2928 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:14.413300 2927 ts_tablet_manager.cc:1428] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:14.413595 2927 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:14.415457 2928 tablet_bootstrap.cc:654] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.415843 2927 tablet_bootstrap.cc:654] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.418627 2929 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:14.418742 2929 ts_tablet_manager.cc:1397] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.007s
I20250904 07:53:14.419279 2929 raft_consensus.cc:357] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.419394 2929 raft_consensus.cc:383] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.419448 2929 raft_consensus.cc:738] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.419624 2929 consensus_queue.cc:260] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.420167 2929 ts_tablet_manager.cc:1428] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:14.420398 2929 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:14.421622 2929 tablet_bootstrap.cc:654] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:14.423039 2929 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:14.423174 2929 ts_tablet_manager.cc:1397] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:14.423746 2929 raft_consensus.cc:357] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.423906 2929 raft_consensus.cc:383] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.423967 2929 raft_consensus.cc:738] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.424156 2929 consensus_queue.cc:260] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.424746 2929 ts_tablet_manager.cc:1428] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:14.425446 2928 tablet_bootstrap.cc:492] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:14.425575 2928 ts_tablet_manager.cc:1397] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20250904 07:53:14.425978 2928 raft_consensus.cc:357] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.426084 2928 raft_consensus.cc:383] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.426138 2928 raft_consensus.cc:738] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.426326 2928 consensus_queue.cc:260] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.426756 2928 ts_tablet_manager.cc:1428] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:14.431284 2927 tablet_bootstrap.cc:492] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:14.431442 2927 ts_tablet_manager.cc:1397] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250904 07:53:14.432165 2927 raft_consensus.cc:357] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.432359 2927 raft_consensus.cc:383] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:14.432449 2927 raft_consensus.cc:738] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:14.432653 2927 consensus_queue.cc:260] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.433298 2927 ts_tablet_manager.cc:1428] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:14.437757 2905 raft_consensus.cc:491] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:14.437880 2905 raft_consensus.cc:513] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.438426 2905 leader_election.cc:290] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:14.438773 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9c55548b294b4895483709aa0cd36d" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:14.438822 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9c55548b294b4895483709aa0cd36d" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:14.438942 2291 raft_consensus.cc:2466] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:14.438941 2464 raft_consensus.cc:2466] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:14.439301 2345 leader_election.cc:304] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:14.439487 2905 raft_consensus.cc:2802] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:14.439549 2905 raft_consensus.cc:491] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:14.439606 2905 raft_consensus.cc:3058] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.441536 2905 raft_consensus.cc:513] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.441967 2905 leader_election.cc:290] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:14.442248 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9c55548b294b4895483709aa0cd36d" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:14.442301 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9c55548b294b4895483709aa0cd36d" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:14.442444 2464 raft_consensus.cc:3058] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.442509 2291 raft_consensus.cc:3058] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.445235 2291 raft_consensus.cc:2466] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:14.445235 2464 raft_consensus.cc:2466] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:14.445575 2345 leader_election.cc:304] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:14.445765 2905 raft_consensus.cc:2802] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:14.446396 2905 raft_consensus.cc:695] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:14.446637 2905 consensus_queue.cc:237] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:14.449693 2169 catalog_manager.cc:5582] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:14.569188 2915 raft_consensus.cc:491] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:14.569370 2915 raft_consensus.cc:513] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.570050 2915 leader_election.cc:290] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:14.570269 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4911b93f2bd1424284b0517f51d64b65" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:14.570451 2291 raft_consensus.cc:2466] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:14.570495 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4911b93f2bd1424284b0517f51d64b65" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:14.570624 2388 raft_consensus.cc:2466] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:14.570833 2425 leader_election.cc:304] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:14.571102 2915 raft_consensus.cc:2802] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:14.571198 2915 raft_consensus.cc:491] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:14.571273 2915 raft_consensus.cc:3058] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.573992 2915 raft_consensus.cc:513] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.574561 2915 leader_election.cc:290] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:14.574831 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4911b93f2bd1424284b0517f51d64b65" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:14.574980 2291 raft_consensus.cc:3058] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.574951 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4911b93f2bd1424284b0517f51d64b65" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:14.575157 2388 raft_consensus.cc:3058] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:14.576953 2291 raft_consensus.cc:2466] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:14.577143 2388 raft_consensus.cc:2466] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:14.577271 2425 leader_election.cc:304] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:14.577502 2915 raft_consensus.cc:2802] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:14.577849 2915 raft_consensus.cc:695] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:14.578060 2915 consensus_queue.cc:237] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:14.580534 2168 catalog_manager.cc:5582] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:14.624039 2388 raft_consensus.cc:1273] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:14.624543 2291 raft_consensus.cc:1273] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:14.624611 2938 consensus_queue.cc:1035] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.625137 2938 consensus_queue.cc:1035] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.625385 2463 raft_consensus.cc:1273] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:14.625394 2292 raft_consensus.cc:1273] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:14.625761 2905 consensus_queue.cc:1035] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:14.626101 2936 consensus_queue.cc:1035] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:14.887530 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 183ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:15.141319 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 436ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:15.393908 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 690ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:15.646868 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:15.652655 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00093e+06us
I20250904 07:53:15.653534 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:15.654280 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:15.663190 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 10a9f44c40154f53987c52fbd68e78d6 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:15.663192 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 10a9f44c40154f53987c52fbd68e78d6 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:15.663662 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a9f44c40154f53987c52fbd68e78d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.663966 2378 tablet_service.cc:1468] Processing CreateTablet for tablet e32aa65f41fe4672a9b2e2916e397a23 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:15.664004 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 10a9f44c40154f53987c52fbd68e78d6 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:15.664254 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32aa65f41fe4672a9b2e2916e397a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.664446 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a9f44c40154f53987c52fbd68e78d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.664994 2278 tablet_service.cc:1468] Processing CreateTablet for tablet e32aa65f41fe4672a9b2e2916e397a23 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:15.663671 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a9f44c40154f53987c52fbd68e78d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.668291 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32aa65f41fe4672a9b2e2916e397a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.669329 2454 tablet_service.cc:1468] Processing CreateTablet for tablet e32aa65f41fe4672a9b2e2916e397a23 (DEFAULT_TABLE table=table.13 [id=fac433b3984c42e08f3f11a5fed09867]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:15.669659 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32aa65f41fe4672a9b2e2916e397a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:15.682693 2962 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:15.686544 2962 tablet_bootstrap.cc:654] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.691310 2963 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:15.692541 2961 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:15.696303 2963 tablet_bootstrap.cc:654] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.698733 2963 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:15.699602 2963 ts_tablet_manager.cc:1397] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250904 07:53:15.699887 2961 tablet_bootstrap.cc:654] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.700366 2963 raft_consensus.cc:357] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.700560 2963 raft_consensus.cc:383] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.700626 2963 raft_consensus.cc:738] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.700812 2963 consensus_queue.cc:260] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.698868 2962 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:15.701162 2962 ts_tablet_manager.cc:1397] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20250904 07:53:15.701784 2963 ts_tablet_manager.cc:1428] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:15.702051 2963 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:15.703054 2961 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:15.703181 2961 ts_tablet_manager.cc:1397] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250904 07:53:15.703649 2963 tablet_bootstrap.cc:654] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.705018 2962 raft_consensus.cc:357] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.705061 2961 raft_consensus.cc:357] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.705240 2962 raft_consensus.cc:383] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.705322 2961 raft_consensus.cc:383] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.705348 2962 raft_consensus.cc:738] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.705439 2961 raft_consensus.cc:738] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.705632 2962 consensus_queue.cc:260] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.705638 2961 consensus_queue.cc:260] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.706321 2961 ts_tablet_manager.cc:1428] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:15.706614 2961 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:15.707144 2962 ts_tablet_manager.cc:1428] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250904 07:53:15.707419 2962 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:15.708418 2961 tablet_bootstrap.cc:654] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.708691 2962 tablet_bootstrap.cc:654] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:15.710222 2961 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:15.711324 2963 tablet_bootstrap.cc:492] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:15.711555 2963 ts_tablet_manager.cc:1397] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250904 07:53:15.713202 2963 raft_consensus.cc:357] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.713577 2961 ts_tablet_manager.cc:1397] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20250904 07:53:15.713718 2963 raft_consensus.cc:383] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.713889 2963 raft_consensus.cc:738] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.714100 2963 consensus_queue.cc:260] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.714255 2961 raft_consensus.cc:357] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.714413 2961 raft_consensus.cc:383] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.714519 2961 raft_consensus.cc:738] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.714645 2963 ts_tablet_manager.cc:1428] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:15.714738 2961 consensus_queue.cc:260] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.714870 2962 tablet_bootstrap.cc:492] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:15.715026 2962 ts_tablet_manager.cc:1397] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20250904 07:53:15.715500 2962 raft_consensus.cc:357] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.715615 2962 raft_consensus.cc:383] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:15.715708 2962 raft_consensus.cc:738] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:15.715888 2962 consensus_queue.cc:260] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.717134 2962 ts_tablet_manager.cc:1428] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:15.717612 2961 ts_tablet_manager.cc:1428] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250904 07:53:15.882210 2938 raft_consensus.cc:491] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:15.882378 2938 raft_consensus.cc:513] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.883085 2938 leader_election.cc:290] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:15.883338 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10a9f44c40154f53987c52fbd68e78d6" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:15.883441 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10a9f44c40154f53987c52fbd68e78d6" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:15.883508 2388 raft_consensus.cc:2466] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:15.883560 2291 raft_consensus.cc:2466] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:15.883905 2427 leader_election.cc:304] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:15.884105 2938 raft_consensus.cc:2802] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:15.884173 2938 raft_consensus.cc:491] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:15.884233 2938 raft_consensus.cc:3058] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.886018 2938 raft_consensus.cc:513] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.886483 2938 leader_election.cc:290] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:15.886751 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10a9f44c40154f53987c52fbd68e78d6" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:15.886852 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10a9f44c40154f53987c52fbd68e78d6" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:15.886901 2388 raft_consensus.cc:3058] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.887003 2291 raft_consensus.cc:3058] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.889210 2291 raft_consensus.cc:2466] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:15.889247 2388 raft_consensus.cc:2466] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:15.889540 2425 leader_election.cc:304] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:15.889873 2938 raft_consensus.cc:2802] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:15.890177 2938 raft_consensus.cc:695] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:15.890441 2938 consensus_queue.cc:237] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:15.892266 2922 raft_consensus.cc:491] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:15.892388 2922 raft_consensus.cc:513] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.893141 2922 leader_election.cc:290] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:15.893546 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e32aa65f41fe4672a9b2e2916e397a23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:15.893704 2388 raft_consensus.cc:2466] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:15.893661 2169 catalog_manager.cc:5582] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:15.894002 2249 leader_election.cc:304] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:15.894213 2922 raft_consensus.cc:2802] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:15.894276 2922 raft_consensus.cc:491] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:15.894356 2922 raft_consensus.cc:3058] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.894630 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e32aa65f41fe4672a9b2e2916e397a23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:15.894781 2464 raft_consensus.cc:2466] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:15.896814 2922 raft_consensus.cc:513] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.897357 2922 leader_election.cc:290] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:15.897631 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e32aa65f41fe4672a9b2e2916e397a23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:15.897727 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e32aa65f41fe4672a9b2e2916e397a23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:15.897794 2388 raft_consensus.cc:3058] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.897846 2464 raft_consensus.cc:3058] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:15.900486 2388 raft_consensus.cc:2466] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:15.900486 2464 raft_consensus.cc:2466] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:15.900909 2247 leader_election.cc:304] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:15.901127 2922 raft_consensus.cc:2802] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:15.901468 2922 raft_consensus.cc:695] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:15.901732 2922 consensus_queue.cc:237] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:15.904559 2169 catalog_manager.cc:5582] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:15.927096 2291 raft_consensus.cc:1273] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:15.927474 2464 raft_consensus.cc:1273] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:15.927474 2388 raft_consensus.cc:1273] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:15.927567 2389 raft_consensus.cc:1273] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:15.927816 2970 consensus_queue.cc:1035] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:15.928020 2938 consensus_queue.cc:1035] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:15.928066 2922 consensus_queue.cc:1035] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:15.928457 2922 consensus_queue.cc:1035] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:16.303480 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 297ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:16.559110 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:16.812372 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 809ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:16.951152 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00074e+06us
I20250904 07:53:16.952293 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:16.952999 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:16.960515 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 36e975e08e9542599e32bcc640a83b2a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:16.960651 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 36e975e08e9542599e32bcc640a83b2a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:16.960918 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e975e08e9542599e32bcc640a83b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.961030 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e975e08e9542599e32bcc640a83b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.961902 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 4de16f56eb074e2a910185b0f63b403a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:16.962189 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de16f56eb074e2a910185b0f63b403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.960918 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 36e975e08e9542599e32bcc640a83b2a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:16.965655 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e975e08e9542599e32bcc640a83b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.965685 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 4de16f56eb074e2a910185b0f63b403a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:16.965997 2992 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:16.966027 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de16f56eb074e2a910185b0f63b403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.967531 2992 tablet_bootstrap.cc:654] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.972366 2994 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:16.972470 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 4de16f56eb074e2a910185b0f63b403a (DEFAULT_TABLE table=table.14 [id=e21248a98b754ff8bd77ca94f4fdf48a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:16.972930 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de16f56eb074e2a910185b0f63b403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:16.973439 2995 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:16.974006 2994 tablet_bootstrap.cc:654] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.975492 2995 tablet_bootstrap.cc:654] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.979108 2992 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:16.979229 2992 ts_tablet_manager.cc:1397] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20250904 07:53:16.980916 2992 raft_consensus.cc:357] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.981048 2992 raft_consensus.cc:383] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.981118 2992 raft_consensus.cc:738] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.981304 2992 consensus_queue.cc:260] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.981419 2994 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:16.981559 2994 ts_tablet_manager.cc:1397] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250904 07:53:16.982141 2994 raft_consensus.cc:357] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.982313 2994 raft_consensus.cc:383] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.982415 2994 raft_consensus.cc:738] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.982435 2992 ts_tablet_manager.cc:1428] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:16.982664 2994 consensus_queue.cc:260] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.982757 2992 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:16.983072 2994 ts_tablet_manager.cc:1428] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:16.983438 2994 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:16.985311 2994 tablet_bootstrap.cc:654] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.986093 2992 tablet_bootstrap.cc:654] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.987545 2995 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:16.987676 2995 ts_tablet_manager.cc:1397] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250904 07:53:16.988306 2995 raft_consensus.cc:357] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.988502 2995 raft_consensus.cc:383] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.989058 2995 raft_consensus.cc:738] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.989279 2995 consensus_queue.cc:260] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.989785 2992 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:16.989822 2995 ts_tablet_manager.cc:1428] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:16.989898 2992 ts_tablet_manager.cc:1397] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250904 07:53:16.990090 2995 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:16.990603 2992 raft_consensus.cc:357] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.990803 2992 raft_consensus.cc:383] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.990893 2992 raft_consensus.cc:738] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.991127 2992 consensus_queue.cc:260] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.991623 2992 ts_tablet_manager.cc:1428] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:16.994036 2995 tablet_bootstrap.cc:654] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:16.994246 2994 tablet_bootstrap.cc:492] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:16.994537 2994 ts_tablet_manager.cc:1397] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20250904 07:53:16.995126 2994 raft_consensus.cc:357] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.995278 2994 raft_consensus.cc:383] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.995340 2994 raft_consensus.cc:738] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.995525 2994 consensus_queue.cc:260] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:16.995643 2995 tablet_bootstrap.cc:492] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:16.995747 2995 ts_tablet_manager.cc:1397] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250904 07:53:16.995946 2994 ts_tablet_manager.cc:1428] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:16.996148 2995 raft_consensus.cc:357] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.996250 2995 raft_consensus.cc:383] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:16.996277 2995 raft_consensus.cc:738] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:16.996394 2995 consensus_queue.cc:260] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:16.996749 2995 ts_tablet_manager.cc:1428] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:17.026839 3000 raft_consensus.cc:491] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:17.026968 3000 raft_consensus.cc:513] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:17.027390 3000 leader_election.cc:290] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:17.027684 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de16f56eb074e2a910185b0f63b403a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:17.027743 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de16f56eb074e2a910185b0f63b403a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:17.027822 2291 raft_consensus.cc:2466] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:17.027920 2388 raft_consensus.cc:2466] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:17.028134 2425 leader_election.cc:304] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:17.028414 3000 raft_consensus.cc:2802] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:17.028491 3000 raft_consensus.cc:491] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:17.028550 3000 raft_consensus.cc:3058] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.030620 3000 raft_consensus.cc:513] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:17.031049 3000 leader_election.cc:290] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:17.031280 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de16f56eb074e2a910185b0f63b403a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:17.031343 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de16f56eb074e2a910185b0f63b403a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:17.031411 2291 raft_consensus.cc:3058] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.031522 2388 raft_consensus.cc:3058] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.033146 2388 raft_consensus.cc:2466] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:17.033207 2291 raft_consensus.cc:2466] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:17.033628 2427 leader_election.cc:304] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:17.033865 3000 raft_consensus.cc:2802] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:17.033998 3000 raft_consensus.cc:695] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:17.034353 3000 consensus_queue.cc:237] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:17.044149 2170 catalog_manager.cc:5582] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:17.118470 2922 raft_consensus.cc:491] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:17.118604 2922 raft_consensus.cc:513] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:17.119091 2922 leader_election.cc:290] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:17.119391 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e975e08e9542599e32bcc640a83b2a" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:17.119453 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e975e08e9542599e32bcc640a83b2a" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:17.119625 2388 raft_consensus.cc:2466] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:17.119632 2464 raft_consensus.cc:2466] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:17.119948 2249 leader_election.cc:304] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:17.120190 2922 raft_consensus.cc:2802] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:17.120277 2922 raft_consensus.cc:491] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:17.120383 2922 raft_consensus.cc:3058] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.122144 2922 raft_consensus.cc:513] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:17.122591 2922 leader_election.cc:290] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:17.122917 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e975e08e9542599e32bcc640a83b2a" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:17.122933 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e975e08e9542599e32bcc640a83b2a" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:17.123062 2388 raft_consensus.cc:3058] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.123062 2464 raft_consensus.cc:3058] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:17.124796 2464 raft_consensus.cc:2466] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:17.124831 2388 raft_consensus.cc:2466] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:17.125155 2247 leader_election.cc:304] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:17.125375 2922 raft_consensus.cc:2802] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:17.125487 2922 raft_consensus.cc:695] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:17.125687 2922 consensus_queue.cc:237] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:17.128326 2168 catalog_manager.cc:5582] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:17.161990 2464 raft_consensus.cc:1273] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:17.161998 2388 raft_consensus.cc:1273] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:17.162097 2291 raft_consensus.cc:1273] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:17.162189 2389 raft_consensus.cc:1273] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:17.162611 3002 consensus_queue.cc:1035] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:17.162768 2922 consensus_queue.cc:1035] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:17.162884 3000 consensus_queue.cc:1035] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:17.163553 3003 consensus_queue.cc:1035] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:17.177949 3012 mvcc.cc:204] Tried to move back new op lower bound from 7196558938771308544 to 7196558938261225472. Current Snapshot: MvccSnapshot[applied={T|T < 7196558938771308544}]
W20250904 07:53:17.467996 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 224ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:17.725021 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 479ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:17.981225 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 736ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:18.190243 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00095e+06us
I20250904 07:53:18.191548 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:18.192564 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:18.204245 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 8c611852cad544fca62c3dae1e16cfa9 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:18.204641 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c611852cad544fca62c3dae1e16cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.205574 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 8c611852cad544fca62c3dae1e16cfa9 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:18.206003 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c611852cad544fca62c3dae1e16cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.209025 3026 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:18.210852 3026 tablet_bootstrap.cc:654] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.215870 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 8c611852cad544fca62c3dae1e16cfa9 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:18.216038 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 157bae761cee4869a60c2b508bab84c1 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:18.216311 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c611852cad544fca62c3dae1e16cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.216944 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 157bae761cee4869a60c2b508bab84c1 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:18.217316 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bae761cee4869a60c2b508bab84c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.221220 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bae761cee4869a60c2b508bab84c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.221228 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 157bae761cee4869a60c2b508bab84c1 (DEFAULT_TABLE table=table.15 [id=eccc3103dcb442d8a918651c02d9da8f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:18.221614 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bae761cee4869a60c2b508bab84c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:18.228073 3026 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:18.228233 3026 ts_tablet_manager.cc:1397] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20250904 07:53:18.229233 3026 raft_consensus.cc:357] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.229478 3026 raft_consensus.cc:383] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.229566 3026 raft_consensus.cc:738] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.229825 3026 consensus_queue.cc:260] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.230727 3026 ts_tablet_manager.cc:1428] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:18.231182 3028 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:18.233291 3028 tablet_bootstrap.cc:654] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.235445 3028 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:18.235598 3028 ts_tablet_manager.cc:1397] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:18.236254 3028 raft_consensus.cc:357] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.236569 3028 raft_consensus.cc:383] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.236660 3028 raft_consensus.cc:738] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.236888 3028 consensus_queue.cc:260] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.239245 3026 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:18.239245 3028 ts_tablet_manager.cc:1428] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:18.239606 3028 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:18.240797 3026 tablet_bootstrap.cc:654] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.243657 3030 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:18.244827 3028 tablet_bootstrap.cc:654] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.245532 3030 tablet_bootstrap.cc:654] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.251328 3026 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:18.251456 3026 ts_tablet_manager.cc:1397] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20250904 07:53:18.252065 3026 raft_consensus.cc:357] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.252218 3026 raft_consensus.cc:383] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.252288 3026 raft_consensus.cc:738] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.252461 3026 consensus_queue.cc:260] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.253139 3026 ts_tablet_manager.cc:1428] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:18.254776 3030 tablet_bootstrap.cc:492] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:18.254858 3030 ts_tablet_manager.cc:1397] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20250904 07:53:18.255398 3030 raft_consensus.cc:357] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.255502 3030 raft_consensus.cc:383] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.255532 3030 raft_consensus.cc:738] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.255684 3030 consensus_queue.cc:260] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.256047 3030 ts_tablet_manager.cc:1428] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:18.256542 3030 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:18.258199 3028 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:18.258419 3028 ts_tablet_manager.cc:1397] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20250904 07:53:18.259244 3028 raft_consensus.cc:357] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.259531 3028 raft_consensus.cc:383] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.259639 3028 raft_consensus.cc:738] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.259900 3028 consensus_queue.cc:260] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.260396 3028 ts_tablet_manager.cc:1428] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:18.260757 3030 tablet_bootstrap.cc:654] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:18.262607 3030 tablet_bootstrap.cc:492] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:18.262728 3030 ts_tablet_manager.cc:1397] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250904 07:53:18.263216 3030 raft_consensus.cc:357] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.263350 3030 raft_consensus.cc:383] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:18.263382 3030 raft_consensus.cc:738] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:18.263535 3030 consensus_queue.cc:260] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.263857 3030 ts_tablet_manager.cc:1428] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:18.345260 3020 raft_consensus.cc:491] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:18.345395 3020 raft_consensus.cc:513] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.345794 3020 leader_election.cc:290] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:18.346110 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "157bae761cee4869a60c2b508bab84c1" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:18.346135 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "157bae761cee4869a60c2b508bab84c1" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:18.346278 2291 raft_consensus.cc:2466] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:18.346314 2464 raft_consensus.cc:2466] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:18.346643 2345 leader_election.cc:304] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:18.347055 3020 raft_consensus.cc:2802] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:18.347141 3020 raft_consensus.cc:491] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:18.347223 3020 raft_consensus.cc:3058] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.349114 3020 raft_consensus.cc:513] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.349610 3020 leader_election.cc:290] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:18.349964 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "157bae761cee4869a60c2b508bab84c1" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:18.349978 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "157bae761cee4869a60c2b508bab84c1" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:18.350140 2464 raft_consensus.cc:3058] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.350140 2291 raft_consensus.cc:3058] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.352559 2291 raft_consensus.cc:2466] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:18.352559 2464 raft_consensus.cc:2466] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:18.352960 2345 leader_election.cc:304] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:18.353178 3020 raft_consensus.cc:2802] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:18.353296 3020 raft_consensus.cc:695] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:18.353521 3020 consensus_queue.cc:237] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:18.356247 2168 catalog_manager.cc:5582] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:18.610873 3000 raft_consensus.cc:491] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:18.611033 3000 raft_consensus.cc:513] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.611549 3000 leader_election.cc:290] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:18.611873 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c611852cad544fca62c3dae1e16cfa9" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:18.611900 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c611852cad544fca62c3dae1e16cfa9" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:18.612012 2389 raft_consensus.cc:2466] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:18.612067 2291 raft_consensus.cc:2466] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:18.612356 2427 leader_election.cc:304] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:18.612553 3000 raft_consensus.cc:2802] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:18.612633 3000 raft_consensus.cc:491] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:18.612721 3000 raft_consensus.cc:3058] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.614750 3000 raft_consensus.cc:513] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.615236 3000 leader_election.cc:290] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:18.615479 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c611852cad544fca62c3dae1e16cfa9" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:18.615559 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c611852cad544fca62c3dae1e16cfa9" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:18.615631 2389 raft_consensus.cc:3058] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.615697 2291 raft_consensus.cc:3058] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:18.617344 2291 raft_consensus.cc:2466] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:18.617694 2425 leader_election.cc:304] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:18.617758 2389 raft_consensus.cc:2466] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:18.618286 3000 raft_consensus.cc:2802] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:18.618716 3000 raft_consensus.cc:695] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:18.618980 3000 consensus_queue.cc:237] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:18.621824 2168 catalog_manager.cc:5582] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:18.681001 2291 raft_consensus.cc:1273] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:18.681202 2389 raft_consensus.cc:1273] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:18.681577 3000 consensus_queue.cc:1035] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:18.683156 3000 consensus_queue.cc:1035] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:18.683930 2464 raft_consensus.cc:1273] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:18.684528 3020 consensus_queue.cc:1035] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:18.685225 2290 raft_consensus.cc:1273] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:18.685807 2955 consensus_queue.cc:1035] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:18.914528 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 155ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:19.169731 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 409ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:19.427623 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 664ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:19.683568 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 922ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:19.707813 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00202e+06us
I20250904 07:53:19.709476 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:19.710166 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:19.718076 2378 tablet_service.cc:1468] Processing CreateTablet for tablet b5d23874cb8c4e58b959fcf93ca357f6 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:19.718238 2454 tablet_service.cc:1468] Processing CreateTablet for tablet b5d23874cb8c4e58b959fcf93ca357f6 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:19.718513 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d23874cb8c4e58b959fcf93ca357f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.718621 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d23874cb8c4e58b959fcf93ca357f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.718755 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 6dad7879f8424cb6a107dfe4e3671b29 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:19.719058 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dad7879f8424cb6a107dfe4e3671b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.723626 3058 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:19.723625 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 6dad7879f8424cb6a107dfe4e3671b29 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:19.723980 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dad7879f8424cb6a107dfe4e3671b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.724136 2278 tablet_service.cc:1468] Processing CreateTablet for tablet b5d23874cb8c4e58b959fcf93ca357f6 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:19.724494 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d23874cb8c4e58b959fcf93ca357f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.725498 3058 tablet_bootstrap.cc:654] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.728281 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 6dad7879f8424cb6a107dfe4e3671b29 (DEFAULT_TABLE table=table.16 [id=c8c7333978264265a54b7a0428c9c2f2]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:19.728730 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dad7879f8424cb6a107dfe4e3671b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:19.732153 3060 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:19.737322 3058 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:19.737514 3058 ts_tablet_manager.cc:1397] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20250904 07:53:19.739100 3058 raft_consensus.cc:357] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.739373 3058 raft_consensus.cc:383] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.739477 3058 raft_consensus.cc:738] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.740155 3059 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:19.740327 3058 consensus_queue.cc:260] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.740945 3058 ts_tablet_manager.cc:1428] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:19.741510 3058 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:19.741827 3060 tablet_bootstrap.cc:654] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.742971 3059 tablet_bootstrap.cc:654] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.742971 3058 tablet_bootstrap.cc:654] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.750746 3060 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:19.750929 3060 ts_tablet_manager.cc:1397] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250904 07:53:19.751672 3060 raft_consensus.cc:357] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.751901 3060 raft_consensus.cc:383] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.751997 3060 raft_consensus.cc:738] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.752219 3060 consensus_queue.cc:260] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.752483 3058 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:19.752796 3058 ts_tablet_manager.cc:1397] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250904 07:53:19.753012 3059 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:19.753118 3059 ts_tablet_manager.cc:1397] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20250904 07:53:19.753454 3058 raft_consensus.cc:357] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.753670 3058 raft_consensus.cc:383] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.753710 3059 raft_consensus.cc:357] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.753783 3058 raft_consensus.cc:738] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.753854 3059 raft_consensus.cc:383] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.753914 3059 raft_consensus.cc:738] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.754035 3058 consensus_queue.cc:260] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.754065 3059 consensus_queue.cc:260] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.754516 3060 ts_tablet_manager.cc:1428] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:19.754796 3060 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:19.755460 3058 ts_tablet_manager.cc:1428] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:19.755712 3059 ts_tablet_manager.cc:1428] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:19.755928 3059 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:19.757929 3060 tablet_bootstrap.cc:654] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.758783 3059 tablet_bootstrap.cc:654] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:19.763913 3059 tablet_bootstrap.cc:492] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:19.764134 3059 ts_tablet_manager.cc:1397] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250904 07:53:19.764726 3059 raft_consensus.cc:357] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.764878 3059 raft_consensus.cc:383] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.764978 3059 raft_consensus.cc:738] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.765172 3059 consensus_queue.cc:260] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.765702 3059 ts_tablet_manager.cc:1428] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:19.765828 3060 tablet_bootstrap.cc:492] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:19.765997 3060 ts_tablet_manager.cc:1397] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250904 07:53:19.766631 3060 raft_consensus.cc:357] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.766800 3060 raft_consensus.cc:383] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:19.766896 3060 raft_consensus.cc:738] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:19.767102 3060 consensus_queue.cc:260] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.767583 3060 ts_tablet_manager.cc:1428] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:19.795814 3036 raft_consensus.cc:491] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:19.795975 3036 raft_consensus.cc:513] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.796593 3036 leader_election.cc:290] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:19.796969 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dad7879f8424cb6a107dfe4e3671b29" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:19.797098 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dad7879f8424cb6a107dfe4e3671b29" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:19.797151 2291 raft_consensus.cc:2466] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:19.797259 2389 raft_consensus.cc:2466] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:19.797611 2425 leader_election.cc:304] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:19.797994 3036 raft_consensus.cc:2802] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:19.798086 3036 raft_consensus.cc:491] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:19.798161 3036 raft_consensus.cc:3058] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:19.801501 3036 raft_consensus.cc:513] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.802068 3036 leader_election.cc:290] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:19.802599 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dad7879f8424cb6a107dfe4e3671b29" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:19.802764 2291 raft_consensus.cc:3058] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:19.802873 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dad7879f8424cb6a107dfe4e3671b29" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:19.803007 2389 raft_consensus.cc:3058] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:19.804682 2389 raft_consensus.cc:2466] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:19.805028 2427 leader_election.cc:304] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:19.805230 3036 raft_consensus.cc:2802] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:19.805640 2291 raft_consensus.cc:2466] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:19.805747 3036 raft_consensus.cc:695] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:19.806041 3036 consensus_queue.cc:237] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:19.809058 2168 catalog_manager.cc:5582] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:19.896368 3048 raft_consensus.cc:491] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:19.896521 3048 raft_consensus.cc:513] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.897012 3048 leader_election.cc:290] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:19.897399 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5d23874cb8c4e58b959fcf93ca357f6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:19.897374 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5d23874cb8c4e58b959fcf93ca357f6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:19.897578 2464 raft_consensus.cc:2466] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:19.897578 2291 raft_consensus.cc:2466] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:19.897940 2345 leader_election.cc:304] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:19.898145 3048 raft_consensus.cc:2802] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:19.898219 3048 raft_consensus.cc:491] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:19.898293 3048 raft_consensus.cc:3058] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:19.904955 3048 raft_consensus.cc:513] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:19.905822 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5d23874cb8c4e58b959fcf93ca357f6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:19.905871 3048 leader_election.cc:290] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:19.906008 2463 raft_consensus.cc:3058] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:19.906237 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5d23874cb8c4e58b959fcf93ca357f6" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:19.906390 2291 raft_consensus.cc:3058] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:20.010598 2291 raft_consensus.cc:2466] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:20.010598 2463 raft_consensus.cc:2466] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:20.011009 2345 leader_election.cc:304] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:20.011256 3048 raft_consensus.cc:2802] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:20.011377 3048 raft_consensus.cc:695] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:20.011595 3048 consensus_queue.cc:237] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:20.014587 2170 catalog_manager.cc:5582] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:20.028086 2291 raft_consensus.cc:1273] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:20.028517 2463 raft_consensus.cc:1273] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:20.028699 3048 consensus_queue.cc:1035] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:20.029166 3065 consensus_queue.cc:1035] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:20.029294 2389 raft_consensus.cc:1273] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:20.029649 2291 raft_consensus.cc:1273] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:20.029784 3036 consensus_queue.cc:1035] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:20.030333 3036 consensus_queue.cc:1035] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:20.341364 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 234ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:20.597604 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 491ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:20.854000 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 748ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:21.050872 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00075e+06us
I20250904 07:53:21.051648 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:21.052386 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:21.061499 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 8f2ac889acc74693bd1816bdf8fe3d23 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:21.061919 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ac889acc74693bd1816bdf8fe3d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.062189 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 3335ab06a4b24194a43832dbc0ac5c51 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:21.062197 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 8f2ac889acc74693bd1816bdf8fe3d23 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:21.062551 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3335ab06a4b24194a43832dbc0ac5c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.062727 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ac889acc74693bd1816bdf8fe3d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.066367 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 3335ab06a4b24194a43832dbc0ac5c51 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:21.066674 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3335ab06a4b24194a43832dbc0ac5c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.067924 3091 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:21.069621 3091 tablet_bootstrap.cc:654] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.071142 3091 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:21.071244 3091 ts_tablet_manager.cc:1397] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:21.071964 3091 raft_consensus.cc:357] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.072160 3091 raft_consensus.cc:383] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.072243 3091 raft_consensus.cc:738] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.072786 3093 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:21.072741 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 3335ab06a4b24194a43832dbc0ac5c51 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:21.073079 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3335ab06a4b24194a43832dbc0ac5c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.072943 3091 consensus_queue.cc:260] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.074350 3093 tablet_bootstrap.cc:654] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.072741 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 8f2ac889acc74693bd1816bdf8fe3d23 (DEFAULT_TABLE table=table.17 [id=a52df097e8a04d7eb9838039b39e689d]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:21.076082 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ac889acc74693bd1816bdf8fe3d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:21.076428 3091 ts_tablet_manager.cc:1428] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250904 07:53:21.076804 3091 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:21.081727 3091 tablet_bootstrap.cc:654] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.090920 3096 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:21.091645 3093 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:21.091773 3093 ts_tablet_manager.cc:1397] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20250904 07:53:21.092428 3093 raft_consensus.cc:357] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.092712 3093 raft_consensus.cc:383] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.092833 3093 raft_consensus.cc:738] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.092577 3096 tablet_bootstrap.cc:654] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.093088 3093 consensus_queue.cc:260] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.093885 3081 raft_consensus.cc:491] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:21.094492 3081 raft_consensus.cc:513] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.094851 3093 ts_tablet_manager.cc:1428] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:21.095057 3093 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:21.095033 3081 leader_election.cc:290] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:21.095639 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:21.095654 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:21.094336 3091 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:21.096105 3091 ts_tablet_manager.cc:1397] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
W20250904 07:53:21.096101 2345 leader_election.cc:343] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Illegal state: must be running to vote when last-logged opid is not known
W20250904 07:53:21.096504 2345 leader_election.cc:343] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657): Illegal state: must be running to vote when last-logged opid is not known
I20250904 07:53:21.096619 2345 leader_election.cc:304] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1047321e14f440c818a8f606e49cb81; no voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837
I20250904 07:53:21.096830 3081 raft_consensus.cc:2747] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 07:53:21.096838 3091 raft_consensus.cc:357] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.097437 3091 raft_consensus.cc:383] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.097494 3091 raft_consensus.cc:738] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.097659 3091 consensus_queue.cc:260] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.098235 3093 tablet_bootstrap.cc:654] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.098829 3091 ts_tablet_manager.cc:1428] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:21.099574 3096 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:21.099684 3096 ts_tablet_manager.cc:1397] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20250904 07:53:21.100250 3096 raft_consensus.cc:357] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.101245 3096 raft_consensus.cc:383] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.101631 3096 raft_consensus.cc:738] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.101887 3096 consensus_queue.cc:260] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.103225 3096 ts_tablet_manager.cc:1428] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:21.103466 3096 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:21.106257 3096 tablet_bootstrap.cc:654] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:21.107613 3096 tablet_bootstrap.cc:492] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:21.107725 3096 ts_tablet_manager.cc:1397] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250904 07:53:21.107824 3093 tablet_bootstrap.cc:492] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:21.107936 3093 ts_tablet_manager.cc:1397] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20250904 07:53:21.108404 3096 raft_consensus.cc:357] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.108556 3096 raft_consensus.cc:383] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.108542 3093 raft_consensus.cc:357] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.108636 3096 raft_consensus.cc:738] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.108702 3093 raft_consensus.cc:383] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:21.108757 3093 raft_consensus.cc:738] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:21.108830 3096 consensus_queue.cc:260] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.108937 3093 consensus_queue.cc:260] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.109310 3096 ts_tablet_manager.cc:1428] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:21.111403 3093 ts_tablet_manager.cc:1428] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:21.115732 3048 raft_consensus.cc:491] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:21.115839 3048 raft_consensus.cc:513] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.116250 3048 leader_election.cc:290] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:21.116526 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3335ab06a4b24194a43832dbc0ac5c51" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:21.116580 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3335ab06a4b24194a43832dbc0ac5c51" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:21.116710 2292 raft_consensus.cc:2466] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:21.116710 2466 raft_consensus.cc:2466] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:21.117105 2345 leader_election.cc:304] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:21.117290 3048 raft_consensus.cc:2802] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:21.117362 3048 raft_consensus.cc:491] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:21.117421 3048 raft_consensus.cc:3058] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.119024 3048 raft_consensus.cc:513] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.119442 3048 leader_election.cc:290] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:21.119760 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3335ab06a4b24194a43832dbc0ac5c51" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:21.119819 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3335ab06a4b24194a43832dbc0ac5c51" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:21.119920 2466 raft_consensus.cc:3058] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.119920 2292 raft_consensus.cc:3058] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.122288 2466 raft_consensus.cc:2466] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:21.122295 2292 raft_consensus.cc:2466] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:21.122601 2345 leader_election.cc:304] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:21.122783 3048 raft_consensus.cc:2802] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:21.122912 3048 raft_consensus.cc:695] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:21.123113 3048 consensus_queue.cc:237] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:21.126058 2168 catalog_manager.cc:5582] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:21.427481 3101 raft_consensus.cc:491] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:21.427636 3101 raft_consensus.cc:513] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.428130 3101 leader_election.cc:290] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:21.428504 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:21.428504 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:21.428718 2389 raft_consensus.cc:2466] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:21.428718 2466 raft_consensus.cc:2466] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:21.429112 2247 leader_election.cc:304] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:21.429414 3101 raft_consensus.cc:2802] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:21.429495 3101 raft_consensus.cc:491] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:21.429596 3101 raft_consensus.cc:3058] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.431428 3101 raft_consensus.cc:513] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.431879 3101 leader_election.cc:290] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:21.432226 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:21.432374 2466 raft_consensus.cc:3058] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.434798 2466 raft_consensus.cc:2466] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:21.435058 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2ac889acc74693bd1816bdf8fe3d23" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:21.435115 2247 leader_election.cc:304] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:21.435222 2389 raft_consensus.cc:3058] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:21.435456 3101 raft_consensus.cc:2802] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:21.435585 3101 raft_consensus.cc:695] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:21.435878 3101 consensus_queue.cc:237] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:21.438035 2389 raft_consensus.cc:2466] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:21.439572 2168 catalog_manager.cc:5582] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:21.528345 2466 raft_consensus.cc:1273] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:21.528445 2389 raft_consensus.cc:1273] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:21.529026 3101 consensus_queue.cc:1035] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:21.529189 2922 consensus_queue.cc:1035] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:21.529516 2292 raft_consensus.cc:1273] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:21.530016 3048 consensus_queue.cc:1035] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:21.530869 2466 raft_consensus.cc:1273] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:21.531370 3081 consensus_queue.cc:1035] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:21.534243 3104 mvcc.cc:204] Tried to move back new op lower bound from 7196558956655779840 to 7196558956284203008. Current Snapshot: MvccSnapshot[applied={T|T < 7196558956655779840}]
W20250904 07:53:21.911602 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 295ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:22.167558 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 551ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:22.421665 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 807ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:22.561020 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00073e+06us
I20250904 07:53:22.561861 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:22.562551 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:22.570792 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 72304b06e03748b69f462d924a5ce342 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:22.571216 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72304b06e03748b69f462d924a5ce342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.571355 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 72304b06e03748b69f462d924a5ce342 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:22.571707 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 72304b06e03748b69f462d924a5ce342 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:22.571790 2454 tablet_service.cc:1468] Processing CreateTablet for tablet c3b8571e45544920a1425933d2bc0e33 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:22.572010 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72304b06e03748b69f462d924a5ce342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.572407 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b8571e45544920a1425933d2bc0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.573300 2278 tablet_service.cc:1468] Processing CreateTablet for tablet c3b8571e45544920a1425933d2bc0e33 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:22.573621 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b8571e45544920a1425933d2bc0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.571713 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72304b06e03748b69f462d924a5ce342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.577792 2378 tablet_service.cc:1468] Processing CreateTablet for tablet c3b8571e45544920a1425933d2bc0e33 (DEFAULT_TABLE table=table.18 [id=fba1c56d4ce64f8eb17fd4a9754d5afe]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:22.578092 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b8571e45544920a1425933d2bc0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:22.584774 3124 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:22.588124 3124 tablet_bootstrap.cc:654] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.592514 3126 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:22.592602 3123 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:22.594005 3126 tablet_bootstrap.cc:654] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.594292 3123 tablet_bootstrap.cc:654] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.595793 3126 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:22.595891 3124 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:22.595906 3126 ts_tablet_manager.cc:1397] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:22.596016 3124 ts_tablet_manager.cc:1397] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20250904 07:53:22.596590 3126 raft_consensus.cc:357] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.596648 3124 raft_consensus.cc:357] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.596750 3126 raft_consensus.cc:383] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.596827 3124 raft_consensus.cc:383] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.596932 3124 raft_consensus.cc:738] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.597126 3124 consensus_queue.cc:260] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.597525 3124 ts_tablet_manager.cc:1428] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:22.597752 3124 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:22.599258 3124 tablet_bootstrap.cc:654] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.596827 3126 raft_consensus.cc:738] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.600261 3126 consensus_queue.cc:260] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.601639 3126 ts_tablet_manager.cc:1428] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250904 07:53:22.601239 3124 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:22.601810 3124 ts_tablet_manager.cc:1397] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250904 07:53:22.602030 3126 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:22.602957 3124 raft_consensus.cc:357] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.603175 3124 raft_consensus.cc:383] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.603271 3124 raft_consensus.cc:738] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.603430 3126 tablet_bootstrap.cc:654] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.603611 3124 consensus_queue.cc:260] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.603992 3123 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:22.604136 3123 ts_tablet_manager.cc:1397] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250904 07:53:22.605347 3124 ts_tablet_manager.cc:1428] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:22.604832 3123 raft_consensus.cc:357] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.606264 3123 raft_consensus.cc:383] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.606335 3123 raft_consensus.cc:738] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.606514 3123 consensus_queue.cc:260] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.608179 3123 ts_tablet_manager.cc:1428] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250904 07:53:22.608455 3123 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:22.610903 3123 tablet_bootstrap.cc:654] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:22.613198 3123 tablet_bootstrap.cc:492] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:22.613407 3123 ts_tablet_manager.cc:1397] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250904 07:53:22.613911 3123 raft_consensus.cc:357] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.614073 3123 raft_consensus.cc:383] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.614141 3123 raft_consensus.cc:738] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.614323 3123 consensus_queue.cc:260] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.614620 3126 tablet_bootstrap.cc:492] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:22.614753 3123 ts_tablet_manager.cc:1428] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:22.614787 3126 ts_tablet_manager.cc:1397] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20250904 07:53:22.615406 3126 raft_consensus.cc:357] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.615566 3126 raft_consensus.cc:383] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:22.615662 3126 raft_consensus.cc:738] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:22.615830 3126 consensus_queue.cc:260] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.616258 3126 ts_tablet_manager.cc:1428] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:22.683346 3114 raft_consensus.cc:491] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:22.683522 3114 raft_consensus.cc:513] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.684334 3114 leader_election.cc:290] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:22.684590 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b8571e45544920a1425933d2bc0e33" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:22.684755 2292 raft_consensus.cc:2466] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:22.684880 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b8571e45544920a1425933d2bc0e33" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:22.685006 2389 raft_consensus.cc:2466] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:22.685149 2425 leader_election.cc:304] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:22.685396 3114 raft_consensus.cc:2802] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:22.685461 3114 raft_consensus.cc:491] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:22.685537 3114 raft_consensus.cc:3058] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.687395 3114 raft_consensus.cc:513] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.687873 3114 leader_election.cc:290] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:22.688153 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b8571e45544920a1425933d2bc0e33" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:22.688213 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b8571e45544920a1425933d2bc0e33" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:22.688302 2292 raft_consensus.cc:3058] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.688349 2389 raft_consensus.cc:3058] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.690135 2292 raft_consensus.cc:2466] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:22.690131 2389 raft_consensus.cc:2466] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:22.690493 2425 leader_election.cc:304] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:22.690722 3114 raft_consensus.cc:2802] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:22.691119 3114 raft_consensus.cc:695] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:22.691363 3114 consensus_queue.cc:237] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:22.694336 2168 catalog_manager.cc:5582] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:22.945531 3114 raft_consensus.cc:491] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:22.945698 3114 raft_consensus.cc:513] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.946312 3114 leader_election.cc:290] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:22.946592 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72304b06e03748b69f462d924a5ce342" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:22.946691 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72304b06e03748b69f462d924a5ce342" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:22.946748 2389 raft_consensus.cc:2466] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:22.946866 2290 raft_consensus.cc:2466] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:22.947103 2427 leader_election.cc:304] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:22.947355 3114 raft_consensus.cc:2802] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:22.947425 3114 raft_consensus.cc:491] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:22.947502 3114 raft_consensus.cc:3058] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.950027 3114 raft_consensus.cc:513] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.950693 3114 leader_election.cc:290] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:22.950877 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72304b06e03748b69f462d924a5ce342" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:22.950909 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72304b06e03748b69f462d924a5ce342" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:22.951010 2389 raft_consensus.cc:3058] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.951042 2290 raft_consensus.cc:3058] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:22.952724 2389 raft_consensus.cc:2466] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:22.952798 2290 raft_consensus.cc:2466] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:22.953091 2427 leader_election.cc:304] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:22.953285 3114 raft_consensus.cc:2802] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:22.953413 3114 raft_consensus.cc:695] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:22.953639 3114 consensus_queue.cc:237] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:22.956128 2170 catalog_manager.cc:5582] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:23.036374 2387 raft_consensus.cc:1273] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:23.036427 2290 raft_consensus.cc:1273] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:23.036795 2389 raft_consensus.cc:1273] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:23.036819 3114 consensus_queue.cc:1035] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:23.037037 3132 consensus_queue.cc:1035] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:23.037165 2290 raft_consensus.cc:1273] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:23.037609 3132 consensus_queue.cc:1035] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:23.037901 3114 consensus_queue.cc:1035] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:23.276973 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 159ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:23.530857 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 413ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:23.787456 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 667ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:24.041640 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 923ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:24.065083 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00085e+06us
I20250904 07:53:24.066171 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:24.066807 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:24.075547 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 0814b7aedf2f4e818d683ba3e0d000e8 (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:24.075636 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 0814b7aedf2f4e818d683ba3e0d000e8 (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:24.075968 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0814b7aedf2f4e818d683ba3e0d000e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.075974 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0814b7aedf2f4e818d683ba3e0d000e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.076149 2280 tablet_service.cc:1468] Processing CreateTablet for tablet f35cfd684d4f4cb1bfd92c5388b1f70c (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:24.076457 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35cfd684d4f4cb1bfd92c5388b1f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.076489 2453 tablet_service.cc:1468] Processing CreateTablet for tablet f35cfd684d4f4cb1bfd92c5388b1f70c (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:24.076797 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35cfd684d4f4cb1bfd92c5388b1f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.077253 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 0814b7aedf2f4e818d683ba3e0d000e8 (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:24.077539 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0814b7aedf2f4e818d683ba3e0d000e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.079653 2378 tablet_service.cc:1468] Processing CreateTablet for tablet f35cfd684d4f4cb1bfd92c5388b1f70c (DEFAULT_TABLE table=table.19 [id=65e039260d5c4b78bdd2e3046077853e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:24.079952 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35cfd684d4f4cb1bfd92c5388b1f70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:24.082523 3159 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:24.083873 3159 tablet_bootstrap.cc:654] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.087386 3161 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:24.090339 3161 tablet_bootstrap.cc:654] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.090721 3159 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:24.090909 3159 ts_tablet_manager.cc:1397] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250904 07:53:24.091667 3162 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:24.092278 3159 raft_consensus.cc:357] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.092787 3159 raft_consensus.cc:383] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.092934 3159 raft_consensus.cc:738] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.093191 3159 consensus_queue.cc:260] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.093463 3162 tablet_bootstrap.cc:654] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.098075 3162 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:24.098201 3162 ts_tablet_manager.cc:1397] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250904 07:53:24.098874 3159 ts_tablet_manager.cc:1428] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20250904 07:53:24.098925 3162 raft_consensus.cc:357] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.099108 3162 raft_consensus.cc:383] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.099184 3162 raft_consensus.cc:738] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.099387 3162 consensus_queue.cc:260] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.099120 3159 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:24.100077 3161 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:24.101080 3159 tablet_bootstrap.cc:654] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.100303 3161 ts_tablet_manager.cc:1397] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250904 07:53:24.102159 3162 ts_tablet_manager.cc:1428] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250904 07:53:24.102389 3162 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:24.103037 3161 raft_consensus.cc:357] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.103211 3161 raft_consensus.cc:383] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.103732 3162 tablet_bootstrap.cc:654] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.103464 3161 raft_consensus.cc:738] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.105203 3159 tablet_bootstrap.cc:492] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:24.105159 3161 consensus_queue.cc:260] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.105330 3159 ts_tablet_manager.cc:1397] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250904 07:53:24.105664 3161 ts_tablet_manager.cc:1428] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:24.105995 3159 raft_consensus.cc:357] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.106076 3161 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:24.106132 3159 raft_consensus.cc:383] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.106177 3159 raft_consensus.cc:738] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.106330 3159 consensus_queue.cc:260] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.106776 3159 ts_tablet_manager.cc:1428] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:24.108242 3161 tablet_bootstrap.cc:654] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:24.109815 3162 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:24.109957 3162 ts_tablet_manager.cc:1397] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250904 07:53:24.110639 3162 raft_consensus.cc:357] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.110812 3162 raft_consensus.cc:383] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.110893 3162 raft_consensus.cc:738] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.111091 3162 consensus_queue.cc:260] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.111923 3162 ts_tablet_manager.cc:1428] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:24.117275 3161 tablet_bootstrap.cc:492] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:24.117372 3161 ts_tablet_manager.cc:1397] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20250904 07:53:24.117874 3161 raft_consensus.cc:357] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.117993 3161 raft_consensus.cc:383] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:24.118026 3161 raft_consensus.cc:738] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:24.118172 3161 consensus_queue.cc:260] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.118531 3161 ts_tablet_manager.cc:1428] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:24.287060 2922 raft_consensus.cc:491] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:24.287230 2922 raft_consensus.cc:513] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.287786 2922 leader_election.cc:290] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:24.288055 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0814b7aedf2f4e818d683ba3e0d000e8" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:24.288232 2389 raft_consensus.cc:2466] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:24.288208 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0814b7aedf2f4e818d683ba3e0d000e8" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:24.288364 2463 raft_consensus.cc:2466] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:24.288602 2249 leader_election.cc:304] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:24.288839 2922 raft_consensus.cc:2802] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:24.288946 2922 raft_consensus.cc:491] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:24.289019 2922 raft_consensus.cc:3058] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.291003 2922 raft_consensus.cc:513] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.291409 2922 leader_election.cc:290] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:24.291719 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0814b7aedf2f4e818d683ba3e0d000e8" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:24.291746 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0814b7aedf2f4e818d683ba3e0d000e8" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:24.291837 2389 raft_consensus.cc:3058] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.291889 2463 raft_consensus.cc:3058] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.293423 2389 raft_consensus.cc:2466] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:24.293467 2463 raft_consensus.cc:2466] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:24.293731 2249 leader_election.cc:304] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:24.293941 2922 raft_consensus.cc:2802] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:24.294051 2922 raft_consensus.cc:695] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:24.294246 2922 consensus_queue.cc:237] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:24.305051 2170 catalog_manager.cc:5582] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:24.414820 2922 raft_consensus.cc:491] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:24.414961 2922 raft_consensus.cc:513] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.415432 2922 leader_election.cc:290] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:24.415763 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f35cfd684d4f4cb1bfd92c5388b1f70c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:24.415761 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f35cfd684d4f4cb1bfd92c5388b1f70c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:24.415948 2389 raft_consensus.cc:2466] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:24.415974 2464 raft_consensus.cc:2466] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:24.416306 2249 leader_election.cc:304] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:24.416543 2922 raft_consensus.cc:2802] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:24.416610 2922 raft_consensus.cc:491] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:24.416687 2922 raft_consensus.cc:3058] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.418951 2922 raft_consensus.cc:513] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.419404 2922 leader_election.cc:290] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:24.419729 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f35cfd684d4f4cb1bfd92c5388b1f70c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:24.419862 2464 raft_consensus.cc:3058] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.420218 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f35cfd684d4f4cb1bfd92c5388b1f70c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:24.420356 2389 raft_consensus.cc:3058] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:24.421949 2464 raft_consensus.cc:2466] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:24.422281 2247 leader_election.cc:304] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:24.422511 2922 raft_consensus.cc:2802] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:24.422607 2922 raft_consensus.cc:695] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:24.422714 2389 raft_consensus.cc:2466] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:24.422813 2922 consensus_queue.cc:237] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:24.426829 2170 catalog_manager.cc:5582] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:24.448251 2464 raft_consensus.cc:1273] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:24.448431 2389 raft_consensus.cc:1273] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:24.449000 2464 raft_consensus.cc:1273] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:24.448949 3163 consensus_queue.cc:1035] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:24.449133 2389 raft_consensus.cc:1273] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:24.449229 2922 consensus_queue.cc:1035] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:24.449527 3163 consensus_queue.cc:1035] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:24.449657 3165 consensus_queue.cc:1035] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:24.697919 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 174ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:24.954304 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 430ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:25.210000 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 686ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:25.465341 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:25.469485 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00096e+06us
I20250904 07:53:25.470547 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:25.471181 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:25.478928 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 479353cf35ca440aa4e7a9c70307a98f (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:25.479362 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479353cf35ca440aa4e7a9c70307a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.479321 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 479353cf35ca440aa4e7a9c70307a98f (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:25.479523 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 422243e6ec0b46c68d96f46423bd8961 (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:25.479655 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479353cf35ca440aa4e7a9c70307a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.479817 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422243e6ec0b46c68d96f46423bd8961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.480202 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 422243e6ec0b46c68d96f46423bd8961 (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:25.480506 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422243e6ec0b46c68d96f46423bd8961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.479665 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 479353cf35ca440aa4e7a9c70307a98f (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:25.481199 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479353cf35ca440aa4e7a9c70307a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.488979 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 422243e6ec0b46c68d96f46423bd8961 (DEFAULT_TABLE table=table.20 [id=8bdab63a8d044aabad95a720b3a81d9e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:25.489303 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422243e6ec0b46c68d96f46423bd8961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:25.496410 3195 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:25.497982 3195 tablet_bootstrap.cc:654] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.499732 3195 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:25.499858 3195 ts_tablet_manager.cc:1397] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250904 07:53:25.500389 3195 raft_consensus.cc:357] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.500532 3195 raft_consensus.cc:383] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.500583 3195 raft_consensus.cc:738] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.500788 3195 consensus_queue.cc:260] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.501282 3195 ts_tablet_manager.cc:1428] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:25.501494 3195 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:25.502817 3195 tablet_bootstrap.cc:654] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.504703 3195 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:25.504827 3195 ts_tablet_manager.cc:1397] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:25.504899 3194 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:25.505600 3195 raft_consensus.cc:357] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.505822 3195 raft_consensus.cc:383] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.505913 3195 raft_consensus.cc:738] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.506134 3195 consensus_queue.cc:260] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.506647 3195 ts_tablet_manager.cc:1428] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:25.506716 3194 tablet_bootstrap.cc:654] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.507997 3196 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:25.508351 3194 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:25.508507 3194 ts_tablet_manager.cc:1397] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:25.509209 3194 raft_consensus.cc:357] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.509424 3194 raft_consensus.cc:383] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.509546 3194 raft_consensus.cc:738] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.509797 3196 tablet_bootstrap.cc:654] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.509771 3194 consensus_queue.cc:260] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.510352 3194 ts_tablet_manager.cc:1428] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:25.510565 3194 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:25.511790 3196 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:25.511934 3196 ts_tablet_manager.cc:1397] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:25.512131 3194 tablet_bootstrap.cc:654] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.512580 3196 raft_consensus.cc:357] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.512753 3196 raft_consensus.cc:383] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.512818 3196 raft_consensus.cc:738] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.513236 3196 consensus_queue.cc:260] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.513945 3196 ts_tablet_manager.cc:1428] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:25.514218 3196 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:25.515463 3196 tablet_bootstrap.cc:654] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:25.516904 3196 tablet_bootstrap.cc:492] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:25.517063 3196 ts_tablet_manager.cc:1397] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:25.517665 3196 raft_consensus.cc:357] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.517781 3196 raft_consensus.cc:383] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.517869 3196 raft_consensus.cc:738] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.518036 3196 consensus_queue.cc:260] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.518354 3196 ts_tablet_manager.cc:1428] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:25.521986 3194 tablet_bootstrap.cc:492] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:25.522094 3194 ts_tablet_manager.cc:1397] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20250904 07:53:25.522650 3194 raft_consensus.cc:357] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.522763 3194 raft_consensus.cc:383] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:25.522825 3194 raft_consensus.cc:738] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:25.523005 3194 consensus_queue.cc:260] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.523308 3194 ts_tablet_manager.cc:1428] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:25.528609 3189 raft_consensus.cc:491] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:25.528733 3189 raft_consensus.cc:513] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.529234 3189 leader_election.cc:290] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:25.529508 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "422243e6ec0b46c68d96f46423bd8961" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:25.529558 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "422243e6ec0b46c68d96f46423bd8961" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:25.529671 2389 raft_consensus.cc:2466] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:25.529697 2466 raft_consensus.cc:2466] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:25.529964 2249 leader_election.cc:304] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:25.530159 3189 raft_consensus.cc:2802] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:25.530234 3189 raft_consensus.cc:491] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:25.530305 3189 raft_consensus.cc:3058] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.532265 3189 raft_consensus.cc:513] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.532967 3189 leader_election.cc:290] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:25.533031 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "422243e6ec0b46c68d96f46423bd8961" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:25.533147 2389 raft_consensus.cc:3058] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.533133 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "422243e6ec0b46c68d96f46423bd8961" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:25.533298 2466 raft_consensus.cc:3058] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.534792 2389 raft_consensus.cc:2466] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:25.535082 2249 leader_election.cc:304] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:25.535286 3189 raft_consensus.cc:2802] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:25.535382 3189 raft_consensus.cc:695] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:25.535584 3189 consensus_queue.cc:237] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:25.535683 2466 raft_consensus.cc:2466] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:25.538375 2169 catalog_manager.cc:5582] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:25.553153 3189 raft_consensus.cc:491] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:25.553361 3189 raft_consensus.cc:513] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.553838 3189 leader_election.cc:290] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:25.554163 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "479353cf35ca440aa4e7a9c70307a98f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:25.554183 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "479353cf35ca440aa4e7a9c70307a98f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:25.554304 2389 raft_consensus.cc:2466] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:25.554313 2466 raft_consensus.cc:2466] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:25.554591 2249 leader_election.cc:304] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:25.554782 3189 raft_consensus.cc:2802] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:25.554855 3189 raft_consensus.cc:491] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:25.554929 3189 raft_consensus.cc:3058] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.556614 3189 raft_consensus.cc:513] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.557077 3189 leader_election.cc:290] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:25.557367 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "479353cf35ca440aa4e7a9c70307a98f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:25.557449 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "479353cf35ca440aa4e7a9c70307a98f" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:25.557555 2389 raft_consensus.cc:3058] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.557614 2466 raft_consensus.cc:3058] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:25.559677 2466 raft_consensus.cc:2466] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:25.560022 2247 leader_election.cc:304] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:25.560016 2389 raft_consensus.cc:2466] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:25.560520 3189 raft_consensus.cc:2802] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:25.560665 3189 raft_consensus.cc:695] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:25.560899 3189 consensus_queue.cc:237] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:25.564025 2169 catalog_manager.cc:5582] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:25.582785 2389 raft_consensus.cc:1273] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:25.582827 2466 raft_consensus.cc:1273] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:25.583170 2463 raft_consensus.cc:1273] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:25.583182 2922 consensus_queue.cc:1035] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:25.583366 2389 raft_consensus.cc:1273] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:25.583662 3208 consensus_queue.cc:1035] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:25.583789 3189 consensus_queue.cc:1035] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:25.584005 3208 consensus_queue.cc:1035] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:25.920464 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 265ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:26.180521 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 520ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:26.451524 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 780ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:26.601619 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00078e+06us
I20250904 07:53:26.602826 2169 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:26.603506 2169 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:26.612298 2379 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd781edbe045e381aa09a6df6e4a1b (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:26.612516 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 2a4522e61ea845658ce3694e9fef670c (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:26.612742 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd781edbe045e381aa09a6df6e4a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.612699 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 2a4522e61ea845658ce3694e9fef670c (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:26.612918 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4522e61ea845658ce3694e9fef670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.613052 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4522e61ea845658ce3694e9fef670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.612982 2453 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd781edbe045e381aa09a6df6e4a1b (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:26.613317 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd781edbe045e381aa09a6df6e4a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.613611 2278 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd781edbe045e381aa09a6df6e4a1b (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:26.613922 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd781edbe045e381aa09a6df6e4a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.618177 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 2a4522e61ea845658ce3694e9fef670c (DEFAULT_TABLE table=table.21 [id=a000321cb81b4c7f860409951a177687]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:26.618510 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4522e61ea845658ce3694e9fef670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:26.631623 3228 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:26.632783 3227 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:26.633740 3228 tablet_bootstrap.cc:654] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.634021 3226 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:26.636251 3227 tablet_bootstrap.cc:654] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.636590 3226 tablet_bootstrap.cc:654] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.639540 3228 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:26.639773 3228 ts_tablet_manager.cc:1397] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20250904 07:53:26.640825 3228 raft_consensus.cc:357] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.641228 3228 raft_consensus.cc:383] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.641352 3228 raft_consensus.cc:738] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.641698 3228 consensus_queue.cc:260] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.642455 3228 ts_tablet_manager.cc:1428] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:26.642788 3228 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:26.644846 3228 tablet_bootstrap.cc:654] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.644940 3226 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:26.645123 3226 ts_tablet_manager.cc:1397] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250904 07:53:26.646055 3227 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:26.645962 3226 raft_consensus.cc:357] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.646266 3227 ts_tablet_manager.cc:1397] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250904 07:53:26.646366 3226 raft_consensus.cc:383] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.646515 3226 raft_consensus.cc:738] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.648551 3226 consensus_queue.cc:260] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.648694 3227 raft_consensus.cc:357] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.648947 3227 raft_consensus.cc:383] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.649039 3227 raft_consensus.cc:738] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.649226 3227 consensus_queue.cc:260] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.649861 3227 ts_tablet_manager.cc:1428] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:26.650187 3227 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:26.651185 3226 ts_tablet_manager.cc:1428] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250904 07:53:26.651592 3226 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:26.652849 3228 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:26.653038 3228 ts_tablet_manager.cc:1397] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250904 07:53:26.653074 3227 tablet_bootstrap.cc:654] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.653895 3228 raft_consensus.cc:357] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.654114 3228 raft_consensus.cc:383] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.654242 3228 raft_consensus.cc:738] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.654495 3228 consensus_queue.cc:260] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.655085 3228 ts_tablet_manager.cc:1428] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:26.657065 3226 tablet_bootstrap.cc:654] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:26.658437 3227 tablet_bootstrap.cc:492] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:26.658610 3227 ts_tablet_manager.cc:1397] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250904 07:53:26.659117 3226 tablet_bootstrap.cc:492] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:26.659286 3226 ts_tablet_manager.cc:1397] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250904 07:53:26.659338 3227 raft_consensus.cc:357] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.659582 3227 raft_consensus.cc:383] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.659677 3227 raft_consensus.cc:738] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.659902 3227 consensus_queue.cc:260] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.659951 3226 raft_consensus.cc:357] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.660347 3226 raft_consensus.cc:383] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:26.660440 3226 raft_consensus.cc:738] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:26.660508 3227 ts_tablet_manager.cc:1428] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:26.660631 3226 consensus_queue.cc:260] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.661289 3226 ts_tablet_manager.cc:1428] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:26.679716 3134 raft_consensus.cc:491] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:26.679831 3134 raft_consensus.cc:513] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.680255 3134 leader_election.cc:290] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:26.680598 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd781edbe045e381aa09a6df6e4a1b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:26.680641 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd781edbe045e381aa09a6df6e4a1b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:26.680783 2463 raft_consensus.cc:2466] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:26.680783 2290 raft_consensus.cc:2466] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:26.681156 2345 leader_election.cc:304] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:26.681382 3134 raft_consensus.cc:2802] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:26.681475 3134 raft_consensus.cc:491] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:26.681545 3134 raft_consensus.cc:3058] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.683193 3134 raft_consensus.cc:513] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.683655 3134 leader_election.cc:290] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:26.683960 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd781edbe045e381aa09a6df6e4a1b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:26.684039 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd781edbe045e381aa09a6df6e4a1b" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:26.684098 2463 raft_consensus.cc:3058] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.684206 2290 raft_consensus.cc:3058] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.686494 2463 raft_consensus.cc:2466] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:26.686795 2290 raft_consensus.cc:2466] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:26.686825 2345 leader_election.cc:304] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:26.687242 3134 raft_consensus.cc:2802] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:26.687505 3134 raft_consensus.cc:695] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:26.687731 3134 consensus_queue.cc:237] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:26.690847 2168 catalog_manager.cc:5582] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:26.888904 3208 raft_consensus.cc:491] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:26.889047 3208 raft_consensus.cc:513] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.889624 3208 leader_election.cc:290] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:26.889814 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a4522e61ea845658ce3694e9fef670c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:26.889979 2389 raft_consensus.cc:2466] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:26.890057 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a4522e61ea845658ce3694e9fef670c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:26.890228 2463 raft_consensus.cc:2466] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:26.890347 2249 leader_election.cc:304] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:26.890552 3208 raft_consensus.cc:2802] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:26.890617 3208 raft_consensus.cc:491] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:26.890684 3208 raft_consensus.cc:3058] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.892480 3208 raft_consensus.cc:513] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.893023 3208 leader_election.cc:290] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:26.893314 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a4522e61ea845658ce3694e9fef670c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:26.893460 2389 raft_consensus.cc:3058] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.893543 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a4522e61ea845658ce3694e9fef670c" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:26.893666 2463 raft_consensus.cc:3058] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:26.895161 2389 raft_consensus.cc:2466] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:26.895561 2249 leader_election.cc:304] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:26.895835 3208 raft_consensus.cc:2802] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:26.895980 3208 raft_consensus.cc:695] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:26.896188 3208 consensus_queue.cc:237] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:26.896245 2463 raft_consensus.cc:2466] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:26.899659 2168 catalog_manager.cc:5582] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:26.934512 2463 raft_consensus.cc:1273] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:26.934767 2389 raft_consensus.cc:1273] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:26.935377 3208 consensus_queue.cc:1035] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:26.935611 3240 consensus_queue.cc:1035] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:26.935890 2290 raft_consensus.cc:1273] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:26.936491 3134 consensus_queue.cc:1035] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:26.936540 2463 raft_consensus.cc:1273] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:26.938834 3134 consensus_queue.cc:1035] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:26.952103 3238 mvcc.cc:204] Tried to move back new op lower bound from 7196558978798166016 to 7196558978648866816. Current Snapshot: MvccSnapshot[applied={T|T < 7196558978798166016}]
W20250904 07:53:27.108721 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 103ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:27.363739 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 358ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:27.617559 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 613ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:27.871693 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 867ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:27.952078 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00101e+06us
I20250904 07:53:27.952827 2169 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:27.953725 2169 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:27.961433 2378 tablet_service.cc:1468] Processing CreateTablet for tablet ae6401dea08d46809de5df0d5358deba (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:27.961606 2453 tablet_service.cc:1468] Processing CreateTablet for tablet ae6401dea08d46809de5df0d5358deba (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:27.961913 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6401dea08d46809de5df0d5358deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.962025 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6401dea08d46809de5df0d5358deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.962677 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 527220ead94f46fe9dac566679a9c804 (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:27.963012 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527220ead94f46fe9dac566679a9c804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.965384 2278 tablet_service.cc:1468] Processing CreateTablet for tablet ae6401dea08d46809de5df0d5358deba (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:27.965703 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6401dea08d46809de5df0d5358deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.966975 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 527220ead94f46fe9dac566679a9c804 (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:27.967427 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527220ead94f46fe9dac566679a9c804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.973755 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 527220ead94f46fe9dac566679a9c804 (DEFAULT_TABLE table=table.22 [id=3cf1d269dca748eea54999082b0f6f86]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:27.974152 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527220ead94f46fe9dac566679a9c804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:27.977082 3257 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:27.978981 3257 tablet_bootstrap.cc:654] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:27.985960 3258 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:27.986382 3257 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:27.986536 3257 ts_tablet_manager.cc:1397] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250904 07:53:27.987231 3257 raft_consensus.cc:357] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:27.987394 3257 raft_consensus.cc:383] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:27.987485 3257 raft_consensus.cc:738] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:27.987704 3257 consensus_queue.cc:260] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:27.988301 3257 ts_tablet_manager.cc:1428] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:27.988829 3257 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:27.990464 3259 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:27.990567 3257 tablet_bootstrap.cc:654] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:27.993510 3259 tablet_bootstrap.cc:654] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:27.994168 3257 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:27.994338 3257 ts_tablet_manager.cc:1397] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250904 07:53:27.994484 3258 tablet_bootstrap.cc:654] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:27.995474 3257 raft_consensus.cc:357] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:27.995947 3257 raft_consensus.cc:383] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:27.996064 3257 raft_consensus.cc:738] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:27.996359 3257 consensus_queue.cc:260] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:27.997144 3257 ts_tablet_manager.cc:1428] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:27.998277 3259 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:27.998435 3259 ts_tablet_manager.cc:1397] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250904 07:53:27.999070 3259 raft_consensus.cc:357] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:27.999277 3259 raft_consensus.cc:383] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:27.999387 3259 raft_consensus.cc:738] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:27.999596 3259 consensus_queue.cc:260] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.000123 3259 ts_tablet_manager.cc:1428] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:28.000465 3259 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:28.002202 3259 tablet_bootstrap.cc:654] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:28.003679 3258 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:28.003839 3258 ts_tablet_manager.cc:1397] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20250904 07:53:28.004544 3258 raft_consensus.cc:357] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.004750 3258 raft_consensus.cc:383] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:28.004832 3258 raft_consensus.cc:738] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:28.006007 3258 consensus_queue.cc:260] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.006574 3258 ts_tablet_manager.cc:1428] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:28.007023 3258 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:28.008195 3259 tablet_bootstrap.cc:492] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:28.008368 3259 ts_tablet_manager.cc:1397] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250904 07:53:28.008597 3258 tablet_bootstrap.cc:654] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:28.009037 3259 raft_consensus.cc:357] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.009178 3259 raft_consensus.cc:383] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:28.009263 3259 raft_consensus.cc:738] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:28.009452 3259 consensus_queue.cc:260] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.009780 3259 ts_tablet_manager.cc:1428] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:28.010823 3258 tablet_bootstrap.cc:492] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:28.011006 3258 ts_tablet_manager.cc:1397] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250904 07:53:28.011713 3258 raft_consensus.cc:357] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.011868 3258 raft_consensus.cc:383] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:28.011924 3258 raft_consensus.cc:738] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:28.012111 3258 consensus_queue.cc:260] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.012712 3258 ts_tablet_manager.cc:1428] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:28.033682 3251 raft_consensus.cc:491] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:28.033816 3251 raft_consensus.cc:513] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.034300 3251 leader_election.cc:290] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:28.034590 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "527220ead94f46fe9dac566679a9c804" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:28.034642 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "527220ead94f46fe9dac566679a9c804" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:28.034777 2389 raft_consensus.cc:2466] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:28.034777 2290 raft_consensus.cc:2466] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:28.035183 2425 leader_election.cc:304] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:28.035377 3251 raft_consensus.cc:2802] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:28.035468 3251 raft_consensus.cc:491] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:28.035533 3251 raft_consensus.cc:3058] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.037199 3251 raft_consensus.cc:513] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.037685 3251 leader_election.cc:290] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:28.037953 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "527220ead94f46fe9dac566679a9c804" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:28.038101 2290 raft_consensus.cc:3058] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.038065 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "527220ead94f46fe9dac566679a9c804" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:28.038213 2389 raft_consensus.cc:3058] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.039768 2290 raft_consensus.cc:2466] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:28.039794 2389 raft_consensus.cc:2466] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:28.040099 2425 leader_election.cc:304] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:28.040325 3251 raft_consensus.cc:2802] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:28.040511 3251 raft_consensus.cc:695] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:28.040771 3251 consensus_queue.cc:237] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:28.044896 2172 catalog_manager.cc:5582] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:28.062753 3249 raft_consensus.cc:491] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:28.062870 3249 raft_consensus.cc:513] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.063300 3249 leader_election.cc:290] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:28.063609 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6401dea08d46809de5df0d5358deba" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:28.063619 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6401dea08d46809de5df0d5358deba" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:28.063761 2463 raft_consensus.cc:2466] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:28.063809 2292 raft_consensus.cc:2466] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:28.064093 2345 leader_election.cc:304] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:28.064312 3249 raft_consensus.cc:2802] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:28.064385 3249 raft_consensus.cc:491] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:28.064450 3249 raft_consensus.cc:3058] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.065944 3249 raft_consensus.cc:513] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.066355 3249 leader_election.cc:290] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:28.066656 2463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6401dea08d46809de5df0d5358deba" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:28.066709 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6401dea08d46809de5df0d5358deba" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:28.066762 2463 raft_consensus.cc:3058] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.066841 2292 raft_consensus.cc:3058] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:28.068616 2463 raft_consensus.cc:2466] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:28.068733 2292 raft_consensus.cc:2466] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:28.069105 2345 leader_election.cc:304] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:28.069357 3249 raft_consensus.cc:2802] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:28.069506 3249 raft_consensus.cc:695] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:28.069770 3249 consensus_queue.cc:237] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:28.073932 2172 catalog_manager.cc:5582] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:28.097667 2292 raft_consensus.cc:1273] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:28.097963 2389 raft_consensus.cc:1273] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:28.098033 2290 raft_consensus.cc:1273] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:28.098225 2463 raft_consensus.cc:1273] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:28.098556 3249 consensus_queue.cc:1035] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:28.098855 3134 consensus_queue.cc:1035] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:28.099318 3221 consensus_queue.cc:1035] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:28.099856 3251 consensus_queue.cc:1035] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:28.327356 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 153ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:28.581696 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 408ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:28.838024 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 663ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:29.094524 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 919ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:29.119984 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00088e+06us
I20250904 07:53:29.120724 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:29.121505 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:29.130383 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 4840a3a32d17474a83157fbf1853c7ce (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:29.130800 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4840a3a32d17474a83157fbf1853c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.131069 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 4840a3a32d17474a83157fbf1853c7ce (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:29.131160 2378 tablet_service.cc:1468] Processing CreateTablet for tablet fd2f3a606d184b6dab7466c1bb848869 (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:29.131382 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4840a3a32d17474a83157fbf1853c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.131495 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2f3a606d184b6dab7466c1bb848869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.133126 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 4840a3a32d17474a83157fbf1853c7ce (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:29.133462 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4840a3a32d17474a83157fbf1853c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.136360 3285 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:29.137923 2453 tablet_service.cc:1468] Processing CreateTablet for tablet fd2f3a606d184b6dab7466c1bb848869 (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:29.138267 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2f3a606d184b6dab7466c1bb848869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.138479 2280 tablet_service.cc:1468] Processing CreateTablet for tablet fd2f3a606d184b6dab7466c1bb848869 (DEFAULT_TABLE table=table.23 [id=ae81c8bc54e34a7da9b210af7b51b57e]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:29.138775 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd2f3a606d184b6dab7466c1bb848869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:29.141263 3286 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:29.144263 3286 tablet_bootstrap.cc:654] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.146081 3285 tablet_bootstrap.cc:654] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.147221 3287 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:29.151582 3287 tablet_bootstrap.cc:654] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.154920 3286 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:29.154945 3285 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:29.155094 3286 ts_tablet_manager.cc:1397] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20250904 07:53:29.155176 3285 ts_tablet_manager.cc:1397] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20250904 07:53:29.156023 3286 raft_consensus.cc:357] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.156244 3286 raft_consensus.cc:383] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.156235 3285 raft_consensus.cc:357] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.156411 3286 raft_consensus.cc:738] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.156482 3285 raft_consensus.cc:383] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.156581 3285 raft_consensus.cc:738] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.156637 3286 consensus_queue.cc:260] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.156886 3285 consensus_queue.cc:260] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.157470 3285 ts_tablet_manager.cc:1428] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:29.157716 3285 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:29.158738 3286 ts_tablet_manager.cc:1428] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 07:53:29.158985 3286 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:29.159420 3285 tablet_bootstrap.cc:654] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.162508 3286 tablet_bootstrap.cc:654] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.164780 3287 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:29.165041 3287 ts_tablet_manager.cc:1397] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.008s
I20250904 07:53:29.166020 3287 raft_consensus.cc:357] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.166231 3287 raft_consensus.cc:383] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.166335 3287 raft_consensus.cc:738] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.166570 3287 consensus_queue.cc:260] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.167200 3287 ts_tablet_manager.cc:1428] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:29.167523 3287 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:29.170428 3287 tablet_bootstrap.cc:654] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:29.173179 3286 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:29.173295 3286 ts_tablet_manager.cc:1397] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20250904 07:53:29.173986 3286 raft_consensus.cc:357] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.174142 3286 raft_consensus.cc:383] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.174209 3286 raft_consensus.cc:738] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.174381 3286 consensus_queue.cc:260] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.176071 3286 ts_tablet_manager.cc:1428] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:29.180258 3285 tablet_bootstrap.cc:492] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:29.180384 3285 ts_tablet_manager.cc:1397] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20250904 07:53:29.181058 3285 raft_consensus.cc:357] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.181231 3285 raft_consensus.cc:383] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.181294 3285 raft_consensus.cc:738] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.181516 3285 consensus_queue.cc:260] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.182237 3285 ts_tablet_manager.cc:1428] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:29.185776 3287 tablet_bootstrap.cc:492] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:29.185890 3287 ts_tablet_manager.cc:1397] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20250904 07:53:29.186513 3287 raft_consensus.cc:357] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.186651 3287 raft_consensus.cc:383] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:29.186697 3287 raft_consensus.cc:738] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:29.186851 3287 consensus_queue.cc:260] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.187268 3287 ts_tablet_manager.cc:1428] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:29.211613 3209 raft_consensus.cc:491] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:29.211761 3209 raft_consensus.cc:513] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.212229 3209 leader_election.cc:290] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:29.212532 2387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2f3a606d184b6dab7466c1bb848869" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:29.212543 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2f3a606d184b6dab7466c1bb848869" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:29.212693 2387 raft_consensus.cc:2466] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:29.212739 2466 raft_consensus.cc:2466] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:29.213060 2249 leader_election.cc:304] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:29.213246 3209 raft_consensus.cc:2802] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:29.213320 3209 raft_consensus.cc:491] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:29.213407 3209 raft_consensus.cc:3058] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.215272 3209 raft_consensus.cc:513] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.215744 3209 leader_election.cc:290] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:29.215991 2387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2f3a606d184b6dab7466c1bb848869" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:29.216110 2387 raft_consensus.cc:3058] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.216091 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd2f3a606d184b6dab7466c1bb848869" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:29.216257 2466 raft_consensus.cc:3058] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.217948 2387 raft_consensus.cc:2466] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:29.218218 2249 leader_election.cc:304] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:29.218436 3209 raft_consensus.cc:2802] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:29.218570 2466 raft_consensus.cc:2466] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:29.219889 3209 raft_consensus.cc:695] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:29.220124 3209 consensus_queue.cc:237] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:29.223551 2168 catalog_manager.cc:5582] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:29.251293 3221 raft_consensus.cc:491] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:29.251410 3221 raft_consensus.cc:513] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.251859 3221 leader_election.cc:290] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:29.252189 2387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4840a3a32d17474a83157fbf1853c7ce" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:29.252223 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4840a3a32d17474a83157fbf1853c7ce" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:29.252312 2387 raft_consensus.cc:2466] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:29.252380 2290 raft_consensus.cc:2466] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:29.252662 2427 leader_election.cc:304] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:29.253014 3221 raft_consensus.cc:2802] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:29.253093 3221 raft_consensus.cc:491] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:29.253178 3221 raft_consensus.cc:3058] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.254922 3221 raft_consensus.cc:513] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
W20250904 07:53:29.255188 2415 tablet.cc:2378] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:53:29.255352 3221 leader_election.cc:290] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:29.255584 2387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4840a3a32d17474a83157fbf1853c7ce" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:29.255698 2387 raft_consensus.cc:3058] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.257069 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4840a3a32d17474a83157fbf1853c7ce" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:29.257200 2290 raft_consensus.cc:3058] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:29.257354 2387 raft_consensus.cc:2466] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:29.257678 2427 leader_election.cc:304] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:29.257879 3221 raft_consensus.cc:2802] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:29.258267 3221 raft_consensus.cc:695] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:29.258529 3221 consensus_queue.cc:237] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:29.259625 2290 raft_consensus.cc:2466] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:29.262492 2172 catalog_manager.cc:5582] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:29.279018 2387 raft_consensus.cc:1273] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:29.279043 2466 raft_consensus.cc:1273] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:29.279345 2389 raft_consensus.cc:1273] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:29.279513 3295 consensus_queue.cc:1035] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:29.279767 2290 raft_consensus.cc:1273] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:29.280382 3295 consensus_queue.cc:1035] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:29.280933 3221 consensus_queue.cc:1035] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:29.281313 3301 consensus_queue.cc:1035] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:29.314674 3300 mvcc.cc:204] Tried to move back new op lower bound from 7196558988402884608 to 7196558988325187584. Current Snapshot: MvccSnapshot[applied={T|T < 7196558988402884608}]
W20250904 07:53:29.553596 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 199ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:29.834013 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 457ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:30.089828 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 738ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:30.297825 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00148e+06us
I20250904 07:53:30.299002 2169 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:30.299711 2169 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:30.309651 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 15f3987a102544a180d8c8e0f34ac6bf (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:30.309794 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 15f3987a102544a180d8c8e0f34ac6bf (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:30.310055 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f3987a102544a180d8c8e0f34ac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.310058 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f3987a102544a180d8c8e0f34ac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.310163 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 15f3987a102544a180d8c8e0f34ac6bf (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:30.310525 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f3987a102544a180d8c8e0f34ac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.310163 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 956895d5d04b41169ec5c465cb0bc42a (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:30.311827 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956895d5d04b41169ec5c465cb0bc42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.313382 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 956895d5d04b41169ec5c465cb0bc42a (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:30.313690 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956895d5d04b41169ec5c465cb0bc42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.327121 3318 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:30.330241 3318 tablet_bootstrap.cc:654] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.330518 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 956895d5d04b41169ec5c465cb0bc42a (DEFAULT_TABLE table=table.24 [id=afcb09fdcf2847e5b6aa708ee83b6130]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:30.331005 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956895d5d04b41169ec5c465cb0bc42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:30.355150 3319 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:30.365224 3319 tablet_bootstrap.cc:654] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.367115 3319 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:30.367251 3319 ts_tablet_manager.cc:1397] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250904 07:53:30.367996 3319 raft_consensus.cc:357] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.368209 3319 raft_consensus.cc:383] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.368266 3319 raft_consensus.cc:738] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.368399 3317 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:30.368434 3319 consensus_queue.cc:260] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.368929 3319 ts_tablet_manager.cc:1428] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:30.369197 3319 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:30.369928 3318 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:30.370048 3318 ts_tablet_manager.cc:1397] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.024s
I20250904 07:53:30.370477 3319 tablet_bootstrap.cc:654] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.372311 3317 tablet_bootstrap.cc:654] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.372341 3319 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:30.370756 3318 raft_consensus.cc:357] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.372535 3318 raft_consensus.cc:383] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.372630 3318 raft_consensus.cc:738] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.372445 3319 ts_tablet_manager.cc:1397] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:30.372921 3318 consensus_queue.cc:260] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.373509 3319 raft_consensus.cc:357] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.373706 3319 raft_consensus.cc:383] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.373793 3319 raft_consensus.cc:738] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.374063 3319 consensus_queue.cc:260] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.374620 3319 ts_tablet_manager.cc:1428] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:30.377351 3318 ts_tablet_manager.cc:1428] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250904 07:53:30.377679 3318 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:30.379400 3318 tablet_bootstrap.cc:654] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.385882 3317 tablet_bootstrap.cc:492] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:30.386025 3317 ts_tablet_manager.cc:1397] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250904 07:53:30.386875 3317 raft_consensus.cc:357] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.387030 3317 raft_consensus.cc:383] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.387081 3317 raft_consensus.cc:738] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.387255 3317 consensus_queue.cc:260] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.387614 3318 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:30.387694 3317 ts_tablet_manager.cc:1428] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:30.387720 3318 ts_tablet_manager.cc:1397] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20250904 07:53:30.387950 3317 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:30.388331 3318 raft_consensus.cc:357] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.388474 3318 raft_consensus.cc:383] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.388521 3318 raft_consensus.cc:738] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.388664 3318 consensus_queue.cc:260] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.390044 3317 tablet_bootstrap.cc:654] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:30.391993 3318 ts_tablet_manager.cc:1428] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:30.401866 3317 tablet_bootstrap.cc:492] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:30.401993 3317 ts_tablet_manager.cc:1397] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20250904 07:53:30.402493 3317 raft_consensus.cc:357] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.402616 3317 raft_consensus.cc:383] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:30.402675 3317 raft_consensus.cc:738] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:30.402851 3317 consensus_queue.cc:260] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.403251 3317 ts_tablet_manager.cc:1428] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:30.476807 3301 raft_consensus.cc:491] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:30.476946 3301 raft_consensus.cc:513] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.477362 3301 leader_election.cc:290] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:30.477665 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "956895d5d04b41169ec5c465cb0bc42a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:30.477711 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "956895d5d04b41169ec5c465cb0bc42a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:30.477802 2292 raft_consensus.cc:2466] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:30.477854 2389 raft_consensus.cc:2466] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:30.478118 2425 leader_election.cc:304] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:30.478309 3301 raft_consensus.cc:2802] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:30.478379 3301 raft_consensus.cc:491] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:30.478451 3301 raft_consensus.cc:3058] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.480072 3301 raft_consensus.cc:513] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.480507 3301 leader_election.cc:290] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:30.480762 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "956895d5d04b41169ec5c465cb0bc42a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:30.480813 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "956895d5d04b41169ec5c465cb0bc42a" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:30.480919 2292 raft_consensus.cc:3058] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.480962 2389 raft_consensus.cc:3058] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.483192 2389 raft_consensus.cc:2466] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:30.483194 2292 raft_consensus.cc:2466] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:30.483515 2425 leader_election.cc:304] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:30.483811 3301 raft_consensus.cc:2802] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:30.484192 3301 raft_consensus.cc:695] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:30.484436 3301 consensus_queue.cc:237] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:30.487339 2170 catalog_manager.cc:5582] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:30.553107 3301 raft_consensus.cc:491] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:30.553244 3301 raft_consensus.cc:513] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.553714 3301 leader_election.cc:290] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:30.553985 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15f3987a102544a180d8c8e0f34ac6bf" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:30.554030 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15f3987a102544a180d8c8e0f34ac6bf" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:30.554172 2389 raft_consensus.cc:2466] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:30.554172 2292 raft_consensus.cc:2466] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:30.554548 2425 leader_election.cc:304] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:30.554775 3301 raft_consensus.cc:2802] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:30.554852 3301 raft_consensus.cc:491] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:30.554920 3301 raft_consensus.cc:3058] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.556483 3301 raft_consensus.cc:513] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.557027 3301 leader_election.cc:290] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:30.557379 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15f3987a102544a180d8c8e0f34ac6bf" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:30.557531 2389 raft_consensus.cc:3058] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.557550 2292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15f3987a102544a180d8c8e0f34ac6bf" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:30.557701 2292 raft_consensus.cc:3058] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:30.559672 2292 raft_consensus.cc:2466] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:30.560042 2389 raft_consensus.cc:2466] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:30.560030 2425 leader_election.cc:304] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:30.560348 3301 raft_consensus.cc:2802] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:30.560515 3301 raft_consensus.cc:695] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:30.560809 3301 consensus_queue.cc:237] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:30.563992 2168 catalog_manager.cc:5582] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:30.583426 2389 raft_consensus.cc:1273] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:30.584000 3332 consensus_queue.cc:1035] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:30.584930 2292 raft_consensus.cc:1273] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:30.586361 2290 raft_consensus.cc:1273] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:30.586789 2388 raft_consensus.cc:1273] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:30.588299 3332 consensus_queue.cc:1035] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250904 07:53:30.588616 3327 consensus_queue.cc:1035] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:30.589004 3332 consensus_queue.cc:1035] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:30.874709 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 223ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:31.136554 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 476ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:31.395460 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 738ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:31.600167 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00101e+06us
I20250904 07:53:31.601850 2169 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:31.602505 2169 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:31.612509 2379 tablet_service.cc:1468] Processing CreateTablet for tablet d8f84bd2ad024dbcb28d5f8befcb6c37 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:31.612962 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f84bd2ad024dbcb28d5f8befcb6c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.612967 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 5be5f30e8f684211b92a3e21486c6ed0 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:31.613332 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be5f30e8f684211b92a3e21486c6ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.614079 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 5be5f30e8f684211b92a3e21486c6ed0 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:31.614413 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be5f30e8f684211b92a3e21486c6ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.617341 2454 tablet_service.cc:1468] Processing CreateTablet for tablet d8f84bd2ad024dbcb28d5f8befcb6c37 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:31.617409 2280 tablet_service.cc:1468] Processing CreateTablet for tablet d8f84bd2ad024dbcb28d5f8befcb6c37 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:31.617698 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f84bd2ad024dbcb28d5f8befcb6c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.617940 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 5be5f30e8f684211b92a3e21486c6ed0 (DEFAULT_TABLE table=table.25 [id=b0d70877d1954e1cbfd5f61fbb58a858]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:31.618263 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be5f30e8f684211b92a3e21486c6ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.617698 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f84bd2ad024dbcb28d5f8befcb6c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:31.635886 3353 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:31.641332 3353 tablet_bootstrap.cc:654] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.645030 3352 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:31.647778 3352 tablet_bootstrap.cc:654] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.649255 3351 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:31.649864 3352 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:31.649991 3352 ts_tablet_manager.cc:1397] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250904 07:53:31.650568 3351 tablet_bootstrap.cc:654] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.650527 3352 raft_consensus.cc:357] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.650812 3352 raft_consensus.cc:383] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.650884 3352 raft_consensus.cc:738] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.651084 3352 consensus_queue.cc:260] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.651515 3352 ts_tablet_manager.cc:1428] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:31.652001 3352 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:31.652037 3353 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:31.652287 3353 ts_tablet_manager.cc:1397] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250904 07:53:31.653852 3352 tablet_bootstrap.cc:654] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.654058 3353 raft_consensus.cc:357] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.654311 3353 raft_consensus.cc:383] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.654431 3353 raft_consensus.cc:738] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.654691 3353 consensus_queue.cc:260] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.655625 3353 ts_tablet_manager.cc:1428] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:31.655921 3353 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:31.661388 3351 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:31.661509 3351 ts_tablet_manager.cc:1397] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250904 07:53:31.662184 3351 raft_consensus.cc:357] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.662348 3351 raft_consensus.cc:383] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.662396 3351 raft_consensus.cc:738] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.662576 3351 consensus_queue.cc:260] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.663010 3351 ts_tablet_manager.cc:1428] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:31.663267 3351 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:31.664251 3352 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:31.664331 3352 ts_tablet_manager.cc:1397] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250904 07:53:31.664752 3351 tablet_bootstrap.cc:654] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.665093 3353 tablet_bootstrap.cc:654] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:31.665015 3352 raft_consensus.cc:357] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.665604 3352 raft_consensus.cc:383] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.665673 3352 raft_consensus.cc:738] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.665845 3352 consensus_queue.cc:260] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.666231 3352 ts_tablet_manager.cc:1428] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:31.667243 3351 tablet_bootstrap.cc:492] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:31.667409 3351 ts_tablet_manager.cc:1397] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250904 07:53:31.668097 3351 raft_consensus.cc:357] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.668283 3351 raft_consensus.cc:383] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.668367 3351 raft_consensus.cc:738] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.668565 3351 consensus_queue.cc:260] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.669173 3351 ts_tablet_manager.cc:1428] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:31.671304 3353 tablet_bootstrap.cc:492] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:31.671500 3353 ts_tablet_manager.cc:1397] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250904 07:53:31.672312 3353 raft_consensus.cc:357] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.672575 3353 raft_consensus.cc:383] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:31.672683 3353 raft_consensus.cc:738] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:31.672940 3353 consensus_queue.cc:260] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.673558 3353 ts_tablet_manager.cc:1428] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:31.683115 3337 raft_consensus.cc:491] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:31.683226 3337 raft_consensus.cc:513] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.683642 3337 leader_election.cc:290] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:31.683997 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8f84bd2ad024dbcb28d5f8befcb6c37" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:31.684145 2389 raft_consensus.cc:2466] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:31.683971 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8f84bd2ad024dbcb28d5f8befcb6c37" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:31.684280 2291 raft_consensus.cc:2466] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:31.684578 2427 leader_election.cc:304] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:31.684834 3337 raft_consensus.cc:2802] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:31.684942 3337 raft_consensus.cc:491] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:31.685014 3337 raft_consensus.cc:3058] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.686671 3337 raft_consensus.cc:513] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.687151 3337 leader_election.cc:290] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:31.687928 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8f84bd2ad024dbcb28d5f8befcb6c37" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:31.688084 2290 raft_consensus.cc:3058] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.689477 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8f84bd2ad024dbcb28d5f8befcb6c37" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:31.689666 2389 raft_consensus.cc:3058] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.690656 2290 raft_consensus.cc:2466] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:31.691038 2425 leader_election.cc:304] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:31.691332 3337 raft_consensus.cc:2802] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:31.691537 3337 raft_consensus.cc:695] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:31.691777 3337 consensus_queue.cc:237] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:31.696053 2389 raft_consensus.cc:2466] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:31.696415 3346 raft_consensus.cc:491] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:31.696535 3346 raft_consensus.cc:513] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.697052 3346 leader_election.cc:290] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:31.697485 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5be5f30e8f684211b92a3e21486c6ed0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:31.697630 2290 raft_consensus.cc:2466] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:31.697690 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5be5f30e8f684211b92a3e21486c6ed0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:31.697860 2389 raft_consensus.cc:2466] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:31.697989 2425 leader_election.cc:304] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:31.698457 3337 raft_consensus.cc:2802] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:31.698536 3337 raft_consensus.cc:491] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:31.698609 3337 raft_consensus.cc:3058] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.699642 2170 catalog_manager.cc:5582] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:31.701036 3337 raft_consensus.cc:513] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.701576 3337 leader_election.cc:290] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:31.701889 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5be5f30e8f684211b92a3e21486c6ed0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:31.702037 2388 raft_consensus.cc:3058] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.702270 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5be5f30e8f684211b92a3e21486c6ed0" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:31.702419 2290 raft_consensus.cc:3058] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:31.703729 2388 raft_consensus.cc:2466] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:31.704030 2427 leader_election.cc:304] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:31.704289 3337 raft_consensus.cc:2802] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:31.704418 3337 raft_consensus.cc:695] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:31.704488 2290 raft_consensus.cc:2466] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:31.704655 3337 consensus_queue.cc:237] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:31.708665 2169 catalog_manager.cc:5582] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:31.727072 2290 raft_consensus.cc:1273] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:31.727453 2291 raft_consensus.cc:1273] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:31.727422 2388 raft_consensus.cc:1273] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:31.727625 2389 raft_consensus.cc:1273] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:31.727972 3337 consensus_queue.cc:1035] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:31.728145 3346 consensus_queue.cc:1035] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:31.728477 3346 consensus_queue.cc:1035] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:31.728904 3337 consensus_queue.cc:1035] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:31.950021 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 136ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:32.205716 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 392ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:32.459400 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 647ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:32.716161 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 901ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:32.759296 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00085e+06us
I20250904 07:53:32.760535 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:32.762336 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:32.769873 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 578c6279e6814dd49de7dae96142b3d5 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:32.770311 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c6279e6814dd49de7dae96142b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.770277 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 578c6279e6814dd49de7dae96142b3d5 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:32.770391 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 578c6279e6814dd49de7dae96142b3d5 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:32.770653 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c6279e6814dd49de7dae96142b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.770653 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c6279e6814dd49de7dae96142b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.770725 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 2d07c349e0d2442085909737fa052037 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:32.771046 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d07c349e0d2442085909737fa052037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.776721 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 2d07c349e0d2442085909737fa052037 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:32.777072 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d07c349e0d2442085909737fa052037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.778115 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 2d07c349e0d2442085909737fa052037 (DEFAULT_TABLE table=table.26 [id=95e23e39431246b4a4def13e0d8db7c1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:32.778412 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d07c349e0d2442085909737fa052037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:32.783499 3384 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:32.786041 3385 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:32.786654 3383 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:32.786342 3384 tablet_bootstrap.cc:654] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.788193 3385 tablet_bootstrap.cc:654] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.790889 3383 tablet_bootstrap.cc:654] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.792608 3384 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:32.792725 3384 ts_tablet_manager.cc:1397] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20250904 07:53:32.793207 3383 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:32.794706 3385 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:32.794822 3383 ts_tablet_manager.cc:1397] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20250904 07:53:32.794852 3385 ts_tablet_manager.cc:1397] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250904 07:53:32.795539 3385 raft_consensus.cc:357] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.795600 3383 raft_consensus.cc:357] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.795730 3385 raft_consensus.cc:383] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.795759 3383 raft_consensus.cc:383] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.795841 3385 raft_consensus.cc:738] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.795858 3383 raft_consensus.cc:738] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.796078 3385 consensus_queue.cc:260] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.796589 3385 ts_tablet_manager.cc:1428] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:32.796095 3383 consensus_queue.cc:260] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.796962 3385 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:32.797202 3383 ts_tablet_manager.cc:1428] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:32.797451 3383 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:32.798424 3385 tablet_bootstrap.cc:654] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.801492 3383 tablet_bootstrap.cc:654] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.803097 3383 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:32.803176 3383 ts_tablet_manager.cc:1397] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20250904 07:53:32.803751 3383 raft_consensus.cc:357] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.803882 3383 raft_consensus.cc:383] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.803936 3383 raft_consensus.cc:738] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.804121 3383 consensus_queue.cc:260] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.793493 3384 raft_consensus.cc:357] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.804304 3384 raft_consensus.cc:383] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.804380 3384 raft_consensus.cc:738] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.804596 3383 ts_tablet_manager.cc:1428] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:32.804656 3384 consensus_queue.cc:260] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.805265 3384 ts_tablet_manager.cc:1428] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250904 07:53:32.805517 3384 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:32.806007 3385 tablet_bootstrap.cc:492] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:32.806154 3385 ts_tablet_manager.cc:1397] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20250904 07:53:32.806836 3385 raft_consensus.cc:357] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.807010 3385 raft_consensus.cc:383] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.807098 3385 raft_consensus.cc:738] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.807137 3384 tablet_bootstrap.cc:654] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:32.807292 3385 consensus_queue.cc:260] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:32.807869 3385 ts_tablet_manager.cc:1428] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:32.809253 3384 tablet_bootstrap.cc:492] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:32.809382 3384 ts_tablet_manager.cc:1397] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:32.810003 3384 raft_consensus.cc:357] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.810181 3384 raft_consensus.cc:383] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:32.810266 3384 raft_consensus.cc:738] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:32.810439 3384 consensus_queue.cc:260] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.810835 3384 ts_tablet_manager.cc:1428] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:32.940023 3249 raft_consensus.cc:491] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:32.940228 3249 raft_consensus.cc:513] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.941128 3249 leader_election.cc:290] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:32.941421 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d07c349e0d2442085909737fa052037" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:32.941606 2291 raft_consensus.cc:2466] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:32.941716 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d07c349e0d2442085909737fa052037" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:32.941928 2464 raft_consensus.cc:2466] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:32.942031 2345 leader_election.cc:304] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:32.942288 3249 raft_consensus.cc:2802] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:32.942409 3249 raft_consensus.cc:491] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:32.942528 3249 raft_consensus.cc:3058] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:32.944737 3249 raft_consensus.cc:513] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.945282 3249 leader_election.cc:290] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:32.945647 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d07c349e0d2442085909737fa052037" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:32.945643 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d07c349e0d2442085909737fa052037" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:32.945852 2464 raft_consensus.cc:3058] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:32.945921 2291 raft_consensus.cc:3058] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:32.947577 2464 raft_consensus.cc:2466] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:32.947705 2291 raft_consensus.cc:2466] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:32.947937 2345 leader_election.cc:304] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:32.948200 3249 raft_consensus.cc:2802] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:32.948354 3249 raft_consensus.cc:695] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:32.948601 3249 consensus_queue.cc:237] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:32.951690 2169 catalog_manager.cc:5582] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:33.008793 3377 raft_consensus.cc:491] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:33.008963 3377 raft_consensus.cc:513] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:33.009514 3377 leader_election.cc:290] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:33.009829 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c6279e6814dd49de7dae96142b3d5" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:33.009909 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c6279e6814dd49de7dae96142b3d5" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:33.010027 2388 raft_consensus.cc:2466] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:33.010087 2291 raft_consensus.cc:2466] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:33.010419 2427 leader_election.cc:304] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:33.010645 3377 raft_consensus.cc:2802] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:33.010744 3377 raft_consensus.cc:491] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:33.010836 3377 raft_consensus.cc:3058] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:33.012487 3377 raft_consensus.cc:513] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:33.013012 3377 leader_election.cc:290] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:33.013262 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c6279e6814dd49de7dae96142b3d5" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:33.013342 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c6279e6814dd49de7dae96142b3d5" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:33.013448 2388 raft_consensus.cc:3058] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:33.013478 2291 raft_consensus.cc:3058] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:33.016080 2291 raft_consensus.cc:2466] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:33.016482 2425 leader_election.cc:304] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:33.016778 3377 raft_consensus.cc:2802] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:33.016958 3377 raft_consensus.cc:695] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:33.017234 3377 consensus_queue.cc:237] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:33.021860 2168 catalog_manager.cc:5582] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:33.034567 2388 raft_consensus.cc:2466] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:33.086814 2464 raft_consensus.cc:1273] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:33.087064 2291 raft_consensus.cc:1273] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:33.087467 2290 raft_consensus.cc:1273] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:33.087540 3249 consensus_queue.cc:1035] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:33.087779 3134 consensus_queue.cc:1035] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:33.087965 3377 consensus_queue.cc:1035] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:33.088349 2388 raft_consensus.cc:1273] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:33.089854 3364 consensus_queue.cc:1035] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:33.371781 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 198ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:33.630348 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 453ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:33.885382 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 712ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:34.119696 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00095e+06us
I20250904 07:53:34.121017 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:34.121719 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:34.140827 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 523c68b1e547434cbc499d3c12e90a21 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:34.141291 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523c68b1e547434cbc499d3c12e90a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.141454 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 523c68b1e547434cbc499d3c12e90a21 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:34.141911 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523c68b1e547434cbc499d3c12e90a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.144268 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 523c68b1e547434cbc499d3c12e90a21 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:34.144608 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523c68b1e547434cbc499d3c12e90a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.144675 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 722589bedbd74a42a0918a9445743358 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:34.145143 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 722589bedbd74a42a0918a9445743358 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:34.145452 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722589bedbd74a42a0918a9445743358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.145803 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722589bedbd74a42a0918a9445743358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.145881 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 722589bedbd74a42a0918a9445743358 (DEFAULT_TABLE table=table.27 [id=1e63a7b559494c0da69d259a05f01eae]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:34.146171 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722589bedbd74a42a0918a9445743358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:34.157269 3416 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:34.159471 3415 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:34.164992 3415 tablet_bootstrap.cc:654] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.165767 3416 tablet_bootstrap.cc:654] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.169338 3415 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:34.169464 3415 ts_tablet_manager.cc:1397] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250904 07:53:34.170198 3417 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:34.170326 3415 raft_consensus.cc:357] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.170603 3415 raft_consensus.cc:383] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.170708 3415 raft_consensus.cc:738] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.170948 3415 consensus_queue.cc:260] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.171617 3416 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:34.171729 3416 ts_tablet_manager.cc:1397] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250904 07:53:34.172473 3415 ts_tablet_manager.cc:1428] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:34.172765 3415 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:34.173277 3416 raft_consensus.cc:357] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.173439 3416 raft_consensus.cc:383] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.173485 3416 raft_consensus.cc:738] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.173625 3416 consensus_queue.cc:260] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.174723 3417 tablet_bootstrap.cc:654] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.175581 3416 ts_tablet_manager.cc:1428] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250904 07:53:34.175858 3416 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:34.177209 3415 tablet_bootstrap.cc:654] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.185617 3416 tablet_bootstrap.cc:654] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.188517 3415 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:34.188728 3415 ts_tablet_manager.cc:1397] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20250904 07:53:34.190285 3415 raft_consensus.cc:357] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.190516 3415 raft_consensus.cc:383] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.190619 3415 raft_consensus.cc:738] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.190869 3415 consensus_queue.cc:260] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.191443 3415 ts_tablet_manager.cc:1428] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250904 07:53:34.195160 3416 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:34.195328 3416 ts_tablet_manager.cc:1397] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20250904 07:53:34.196028 3416 raft_consensus.cc:357] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.196189 3416 raft_consensus.cc:383] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.196250 3416 raft_consensus.cc:738] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.198102 3417 tablet_bootstrap.cc:492] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:34.198217 3417 ts_tablet_manager.cc:1397] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20250904 07:53:34.198078 3416 consensus_queue.cc:260] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.198647 3416 ts_tablet_manager.cc:1428] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:34.198908 3417 raft_consensus.cc:357] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.199071 3417 raft_consensus.cc:383] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.199126 3417 raft_consensus.cc:738] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.199308 3417 consensus_queue.cc:260] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.199891 3417 ts_tablet_manager.cc:1428] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:34.200285 3417 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:34.201983 3417 tablet_bootstrap.cc:654] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:34.204017 3417 tablet_bootstrap.cc:492] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:34.204232 3417 ts_tablet_manager.cc:1397] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:53:34.205005 3417 raft_consensus.cc:357] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.205235 3417 raft_consensus.cc:383] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:34.205436 3417 raft_consensus.cc:738] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:34.205659 3417 consensus_queue.cc:260] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.206037 3417 ts_tablet_manager.cc:1428] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:34.364236 3377 raft_consensus.cc:491] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:34.364398 3377 raft_consensus.cc:513] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.365037 3377 leader_election.cc:290] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:34.365442 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "523c68b1e547434cbc499d3c12e90a21" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:34.365504 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "523c68b1e547434cbc499d3c12e90a21" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:34.365597 2291 raft_consensus.cc:2466] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:34.365656 2388 raft_consensus.cc:2466] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:34.365955 2425 leader_election.cc:304] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:34.366318 3377 raft_consensus.cc:2802] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:34.366407 3377 raft_consensus.cc:491] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:34.366477 3377 raft_consensus.cc:3058] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.368603 3377 raft_consensus.cc:513] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.369064 3377 leader_election.cc:290] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697)
I20250904 07:53:34.369390 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "523c68b1e547434cbc499d3c12e90a21" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:34.369422 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "523c68b1e547434cbc499d3c12e90a21" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:34.369504 2291 raft_consensus.cc:3058] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.369550 2388 raft_consensus.cc:3058] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.371155 2291 raft_consensus.cc:2466] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:34.371470 2425 leader_election.cc:304] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:34.371475 2388 raft_consensus.cc:2466] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:34.371671 3377 raft_consensus.cc:2802] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:34.371796 3377 raft_consensus.cc:695] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:34.371996 3377 consensus_queue.cc:237] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:34.376252 2170 catalog_manager.cc:5582] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:34.530153 3249 raft_consensus.cc:491] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:34.530301 3249 raft_consensus.cc:513] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.530804 3249 leader_election.cc:290] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:34.531114 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "722589bedbd74a42a0918a9445743358" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:34.531265 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "722589bedbd74a42a0918a9445743358" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:34.531296 2290 raft_consensus.cc:2466] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:34.531396 2466 raft_consensus.cc:2466] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:34.531663 2345 leader_election.cc:304] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:34.531873 3249 raft_consensus.cc:2802] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:34.531962 3249 raft_consensus.cc:491] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:34.532018 3249 raft_consensus.cc:3058] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.533944 3249 raft_consensus.cc:513] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.534401 3249 leader_election.cc:290] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:34.534685 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "722589bedbd74a42a0918a9445743358" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:34.534770 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "722589bedbd74a42a0918a9445743358" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:34.534832 2290 raft_consensus.cc:3058] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.534919 2466 raft_consensus.cc:3058] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:34.537146 2466 raft_consensus.cc:2466] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:34.537547 2290 raft_consensus.cc:2466] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:34.538358 2345 leader_election.cc:304] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:34.538570 3249 raft_consensus.cc:2802] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:34.538694 3249 raft_consensus.cc:695] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:34.538954 3249 consensus_queue.cc:237] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:34.541700 2170 catalog_manager.cc:5582] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:34.601006 2466 raft_consensus.cc:1273] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:34.601771 2290 raft_consensus.cc:1273] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:34.601874 2388 raft_consensus.cc:1273] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:34.601969 2291 raft_consensus.cc:1273] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:34.602118 3249 consensus_queue.cc:1035] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:34.602309 3377 consensus_queue.cc:1035] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:34.602456 3408 consensus_queue.cc:1035] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:34.602552 3410 consensus_queue.cc:1035] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:34.617420 3427 mvcc.cc:204] Tried to move back new op lower bound from 7196559010205339648 to 7196559009270001664. Current Snapshot: MvccSnapshot[applied={T|T < 7196559010205339648}]
W20250904 07:53:34.846923 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 167ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:35.105854 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:35.362200 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 682ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:35.616343 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 938ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:35.625082 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00093e+06us
I20250904 07:53:35.625830 2168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:35.626603 2168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:35.637492 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 0b8a122459d34c1494e5f8bf2a0c92b1 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:35.637693 2454 tablet_service.cc:1468] Processing CreateTablet for tablet 0b8a122459d34c1494e5f8bf2a0c92b1 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:35.637928 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8a122459d34c1494e5f8bf2a0c92b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.638073 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8a122459d34c1494e5f8bf2a0c92b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.638818 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 64ee60c907814f64858eb85b6f555684 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:35.639122 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ee60c907814f64858eb85b6f555684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.641189 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 64ee60c907814f64858eb85b6f555684 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:35.641525 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ee60c907814f64858eb85b6f555684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.642617 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 64ee60c907814f64858eb85b6f555684 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:35.642911 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ee60c907814f64858eb85b6f555684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.645758 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 0b8a122459d34c1494e5f8bf2a0c92b1 (DEFAULT_TABLE table=table.28 [id=8a0986eea9a247a08026a7b859bee09a]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:35.646086 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8a122459d34c1494e5f8bf2a0c92b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:35.646232 3442 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:35.648554 3442 tablet_bootstrap.cc:654] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.648694 3443 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:35.651525 3443 tablet_bootstrap.cc:654] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.655259 3442 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:35.655259 3443 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:35.655414 3442 ts_tablet_manager.cc:1397] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250904 07:53:35.655414 3443 ts_tablet_manager.cc:1397] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250904 07:53:35.656137 3442 raft_consensus.cc:357] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.656312 3442 raft_consensus.cc:383] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.656364 3442 raft_consensus.cc:738] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.656558 3442 consensus_queue.cc:260] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.657035 3442 ts_tablet_manager.cc:1428] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:35.657526 3442 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:35.658999 3442 tablet_bootstrap.cc:654] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.660737 3446 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:35.660846 3443 raft_consensus.cc:357] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.661388 3443 raft_consensus.cc:383] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.662786 3443 raft_consensus.cc:738] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.663111 3446 tablet_bootstrap.cc:654] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.663028 3443 consensus_queue.cc:260] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.663811 3443 ts_tablet_manager.cc:1428] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250904 07:53:35.664443 3443 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:35.665452 3442 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:35.665570 3442 ts_tablet_manager.cc:1397] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250904 07:53:35.665953 3443 tablet_bootstrap.cc:654] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.666112 3442 raft_consensus.cc:357] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.666234 3442 raft_consensus.cc:383] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.666296 3442 raft_consensus.cc:738] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.666488 3442 consensus_queue.cc:260] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.666947 3442 ts_tablet_manager.cc:1428] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:35.669330 3443 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:35.669438 3443 ts_tablet_manager.cc:1397] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:53:35.669517 3446 tablet_bootstrap.cc:492] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:35.669661 3446 ts_tablet_manager.cc:1397] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250904 07:53:35.670054 3443 raft_consensus.cc:357] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.670177 3443 raft_consensus.cc:383] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.670275 3443 raft_consensus.cc:738] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.670389 3446 raft_consensus.cc:357] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.670562 3446 raft_consensus.cc:383] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.670631 3446 raft_consensus.cc:738] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.670862 3446 consensus_queue.cc:260] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.671373 3446 ts_tablet_manager.cc:1428] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:35.671705 3446 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:35.673588 3446 tablet_bootstrap.cc:654] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:35.670464 3443 consensus_queue.cc:260] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.675145 3443 ts_tablet_manager.cc:1428] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250904 07:53:35.676733 3446 tablet_bootstrap.cc:492] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:35.676872 3446 ts_tablet_manager.cc:1397] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250904 07:53:35.677488 3446 raft_consensus.cc:357] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.677662 3446 raft_consensus.cc:383] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:35.677721 3446 raft_consensus.cc:738] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:35.677829 3249 raft_consensus.cc:491] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:35.677949 3249 raft_consensus.cc:513] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.678014 3446 consensus_queue.cc:260] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.678499 3249 leader_election.cc:290] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:35.678826 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ee60c907814f64858eb85b6f555684" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:35.678969 2466 raft_consensus.cc:2466] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:35.679042 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ee60c907814f64858eb85b6f555684" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:35.679157 2290 raft_consensus.cc:2466] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:35.679335 2345 leader_election.cc:304] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:35.679540 3249 raft_consensus.cc:2802] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:35.679606 3249 raft_consensus.cc:491] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:35.679664 3249 raft_consensus.cc:3058] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.678956 3446 ts_tablet_manager.cc:1428] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:35.681619 3249 raft_consensus.cc:513] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.682233 3249 leader_election.cc:290] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:35.682567 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ee60c907814f64858eb85b6f555684" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:35.682689 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ee60c907814f64858eb85b6f555684" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:35.682695 2290 raft_consensus.cc:3058] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.682864 2466 raft_consensus.cc:3058] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.685110 2290 raft_consensus.cc:2466] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:35.685468 2345 leader_election.cc:304] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:35.685110 2466 raft_consensus.cc:2466] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:35.685735 3410 raft_consensus.cc:2802] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:35.685825 3410 raft_consensus.cc:695] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:35.686033 3410 consensus_queue.cc:237] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:35.691545 2170 catalog_manager.cc:5582] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:35.821897 3378 raft_consensus.cc:491] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:35.822034 3378 raft_consensus.cc:513] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.822732 3378 leader_election.cc:290] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:35.823040 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b8a122459d34c1494e5f8bf2a0c92b1" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:35.823108 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b8a122459d34c1494e5f8bf2a0c92b1" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:35.823254 2466 raft_consensus.cc:2466] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:35.823254 2388 raft_consensus.cc:2466] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:35.823604 2247 leader_election.cc:304] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:35.823830 3378 raft_consensus.cc:2802] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:35.823904 3378 raft_consensus.cc:491] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:35.823951 3378 raft_consensus.cc:3058] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.826166 3378 raft_consensus.cc:513] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.826829 3378 leader_election.cc:290] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:35.827087 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b8a122459d34c1494e5f8bf2a0c92b1" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:35.827243 2388 raft_consensus.cc:3058] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.827189 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b8a122459d34c1494e5f8bf2a0c92b1" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:35.827385 2466 raft_consensus.cc:3058] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:35.829211 2388 raft_consensus.cc:2466] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:35.829217 2466 raft_consensus.cc:2466] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:35.829576 2249 leader_election.cc:304] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:35.829931 3378 raft_consensus.cc:2802] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:35.830046 3378 raft_consensus.cc:695] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:35.830291 3378 consensus_queue.cc:237] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:35.833621 2170 catalog_manager.cc:5582] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:35.853060 2388 raft_consensus.cc:1273] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:35.853052 2466 raft_consensus.cc:1273] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:35.853636 3321 consensus_queue.cc:1035] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:35.853688 2466 raft_consensus.cc:1273] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:35.854132 3321 consensus_queue.cc:1035] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:35.854681 3410 consensus_queue.cc:1035] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:35.855664 2290 raft_consensus.cc:1273] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:35.856144 3410 consensus_queue.cc:1035] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:36.076505 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 134ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:36.335108 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 394ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:36.590376 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 653ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:36.848713 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 908ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:36.883463 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00095e+06us
I20250904 07:53:36.884413 2172 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:36.885129 2172 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:36.896406 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 5422efe674664a778e97827177ba6482 (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:36.896680 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 5422efe674664a778e97827177ba6482 (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:36.896811 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5422efe674664a778e97827177ba6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.896999 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5422efe674664a778e97827177ba6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.896430 2378 tablet_service.cc:1468] Processing CreateTablet for tablet c75cefc8a85947cbbae7977ef83e3eff (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:36.901341 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75cefc8a85947cbbae7977ef83e3eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.901835 2454 tablet_service.cc:1468] Processing CreateTablet for tablet c75cefc8a85947cbbae7977ef83e3eff (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:36.902099 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75cefc8a85947cbbae7977ef83e3eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.902844 3473 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:36.905845 3473 tablet_bootstrap.cc:654] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.906263 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 5422efe674664a778e97827177ba6482 (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:36.906579 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5422efe674664a778e97827177ba6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.909277 2278 tablet_service.cc:1468] Processing CreateTablet for tablet c75cefc8a85947cbbae7977ef83e3eff (DEFAULT_TABLE table=table.29 [id=7ab2dc9053554aaeacfca22c4a49f68f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:36.910960 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75cefc8a85947cbbae7977ef83e3eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:36.913249 3474 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:36.914505 3474 tablet_bootstrap.cc:654] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.916110 3473 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:36.916158 3474 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:36.916229 3473 ts_tablet_manager.cc:1397] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20250904 07:53:36.916326 3474 ts_tablet_manager.cc:1397] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:36.916792 3473 raft_consensus.cc:357] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.916986 3473 raft_consensus.cc:383] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.916965 3474 raft_consensus.cc:357] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.917064 3473 raft_consensus.cc:738] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.917111 3474 raft_consensus.cc:383] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.917214 3474 raft_consensus.cc:738] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.917251 3473 consensus_queue.cc:260] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.917407 3474 consensus_queue.cc:260] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.917709 3473 ts_tablet_manager.cc:1428] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:36.917913 3474 ts_tablet_manager.cc:1428] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:36.918133 3474 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:36.919639 3474 tablet_bootstrap.cc:654] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.930352 3473 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:36.933135 3473 tablet_bootstrap.cc:654] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.936218 3473 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:36.936425 3473 ts_tablet_manager.cc:1397] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250904 07:53:36.938076 3473 raft_consensus.cc:357] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.938354 3473 raft_consensus.cc:383] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.938464 3473 raft_consensus.cc:738] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.938799 3478 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:36.938731 3473 consensus_queue.cc:260] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.939565 3473 ts_tablet_manager.cc:1428] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:36.942682 3478 tablet_bootstrap.cc:654] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.944581 3474 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:36.944749 3474 ts_tablet_manager.cc:1397] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.013s
I20250904 07:53:36.945473 3474 raft_consensus.cc:357] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.945694 3474 raft_consensus.cc:383] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.945784 3474 raft_consensus.cc:738] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.945995 3474 consensus_queue.cc:260] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.946368 3474 ts_tablet_manager.cc:1428] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:36.946518 3478 tablet_bootstrap.cc:492] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:36.946637 3478 ts_tablet_manager.cc:1397] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250904 07:53:36.947252 3478 raft_consensus.cc:357] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.947419 3478 raft_consensus.cc:383] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.947511 3478 raft_consensus.cc:738] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.947747 3478 consensus_queue.cc:260] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:36.948328 3478 ts_tablet_manager.cc:1428] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:36.948839 3478 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:36.950783 3478 tablet_bootstrap.cc:654] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:36.952395 3478 tablet_bootstrap.cc:492] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:36.952560 3478 ts_tablet_manager.cc:1397] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:53:36.953161 3478 raft_consensus.cc:357] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.953310 3478 raft_consensus.cc:383] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:36.953399 3478 raft_consensus.cc:738] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:36.953644 3465 raft_consensus.cc:491] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:36.953572 3478 consensus_queue.cc:260] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.953764 3465 raft_consensus.cc:513] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.954142 3478 ts_tablet_manager.cc:1428] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:36.954290 3465 leader_election.cc:290] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:36.954962 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c75cefc8a85947cbbae7977ef83e3eff" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:36.955133 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c75cefc8a85947cbbae7977ef83e3eff" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:36.955134 2291 raft_consensus.cc:2466] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:36.955340 2464 raft_consensus.cc:2466] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:36.955600 2345 leader_election.cc:304] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:36.956049 3468 raft_consensus.cc:2802] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:36.956128 3468 raft_consensus.cc:491] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:36.956195 3468 raft_consensus.cc:3058] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:36.958679 3468 raft_consensus.cc:513] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.959133 3468 leader_election.cc:290] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:36.959446 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c75cefc8a85947cbbae7977ef83e3eff" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:36.959651 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c75cefc8a85947cbbae7977ef83e3eff" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:36.960291 2464 raft_consensus.cc:3058] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:36.960291 2290 raft_consensus.cc:3058] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:36.962726 2464 raft_consensus.cc:2466] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:36.962726 2290 raft_consensus.cc:2466] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:36.963389 2345 leader_election.cc:304] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:36.963599 3465 raft_consensus.cc:2802] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:36.963732 3465 raft_consensus.cc:695] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:36.963961 3465 consensus_queue.cc:237] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:36.967025 2172 catalog_manager.cc:5582] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:37.162911 3464 raft_consensus.cc:491] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:37.163044 3464 raft_consensus.cc:513] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:37.163574 3464 leader_election.cc:290] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:37.163848 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5422efe674664a778e97827177ba6482" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:37.163877 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5422efe674664a778e97827177ba6482" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:37.164017 2464 raft_consensus.cc:2466] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:37.164017 2389 raft_consensus.cc:2466] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:37.164379 2249 leader_election.cc:304] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:37.164593 3464 raft_consensus.cc:2802] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:37.164686 3464 raft_consensus.cc:491] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:37.164743 3464 raft_consensus.cc:3058] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:37.166574 3464 raft_consensus.cc:513] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:37.167124 3464 leader_election.cc:290] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:37.167387 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5422efe674664a778e97827177ba6482" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:37.167460 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5422efe674664a778e97827177ba6482" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:37.167524 2389 raft_consensus.cc:3058] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:37.167632 2464 raft_consensus.cc:3058] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:37.170214 2389 raft_consensus.cc:2466] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:37.170346 2464 raft_consensus.cc:2466] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:37.170603 2249 leader_election.cc:304] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:37.170805 3464 raft_consensus.cc:2802] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:37.170955 3464 raft_consensus.cc:695] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:37.171213 3464 consensus_queue.cc:237] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:37.174742 2170 catalog_manager.cc:5582] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:37.215059 2464 raft_consensus.cc:1273] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:37.215464 2290 raft_consensus.cc:1273] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:37.216154 3465 consensus_queue.cc:1035] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:37.216289 3468 consensus_queue.cc:1035] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:37.216513 2389 raft_consensus.cc:1273] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:37.216588 2464 raft_consensus.cc:1273] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:37.216936 3466 consensus_queue.cc:1035] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:37.217162 3464 consensus_queue.cc:1035] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:37.255630 3486 mvcc.cc:204] Tried to move back new op lower bound from 7196559020912254976 to 7196559020735365120. Current Snapshot: MvccSnapshot[applied={T|T < 7196559020912254976}]
W20250904 07:53:37.506850 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 203ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:37.763825 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 460ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:38.020615 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 717ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:38.248066 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.65:46697, user_credentials={real_user=slave}} blocked reactor thread for 1.00095e+06us
I20250904 07:53:38.249145 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:38.249826 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:38.261086 2379 tablet_service.cc:1468] Processing CreateTablet for tablet c2381750df7a49288423e2170a7c0838 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:38.261176 2378 tablet_service.cc:1468] Processing CreateTablet for tablet ccd732bc8933444cb35bd041088b2286 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:38.261451 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2381750df7a49288423e2170a7c0838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.261655 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd732bc8933444cb35bd041088b2286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.262395 2454 tablet_service.cc:1468] Processing CreateTablet for tablet c2381750df7a49288423e2170a7c0838 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:38.262688 2454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2381750df7a49288423e2170a7c0838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.265615 2278 tablet_service.cc:1468] Processing CreateTablet for tablet ccd732bc8933444cb35bd041088b2286 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:38.265620 2453 tablet_service.cc:1468] Processing CreateTablet for tablet ccd732bc8933444cb35bd041088b2286 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:38.265960 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd732bc8933444cb35bd041088b2286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.265991 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd732bc8933444cb35bd041088b2286. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.265620 2280 tablet_service.cc:1468] Processing CreateTablet for tablet c2381750df7a49288423e2170a7c0838 (DEFAULT_TABLE table=table.30 [id=15072bcebc314d3fa3b2d58f3b45382f]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:38.269577 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2381750df7a49288423e2170a7c0838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:38.274631 3506 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:38.278136 3506 tablet_bootstrap.cc:654] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.283171 3505 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:38.285053 3505 tablet_bootstrap.cc:654] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.286011 3507 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:38.287897 3507 tablet_bootstrap.cc:654] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.297139 3506 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:38.297292 3506 ts_tablet_manager.cc:1397] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20250904 07:53:38.297829 3507 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:38.297984 3507 ts_tablet_manager.cc:1397] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250904 07:53:38.297847 3506 raft_consensus.cc:357] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.298069 3506 raft_consensus.cc:383] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.298152 3506 raft_consensus.cc:738] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.298426 3506 consensus_queue.cc:260] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.298789 3505 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:38.298938 3505 ts_tablet_manager.cc:1397] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20250904 07:53:38.299582 3505 raft_consensus.cc:357] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.299799 3505 raft_consensus.cc:383] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.299885 3505 raft_consensus.cc:738] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.300102 3505 consensus_queue.cc:260] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.300549 3506 ts_tablet_manager.cc:1428] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:38.300913 3506 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:38.300980 3505 ts_tablet_manager.cc:1428] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:38.302503 3506 tablet_bootstrap.cc:654] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.309432 3505 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:38.299343 3507 raft_consensus.cc:357] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.309804 3507 raft_consensus.cc:383] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.309911 3507 raft_consensus.cc:738] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.310133 3507 consensus_queue.cc:260] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.310679 3507 ts_tablet_manager.cc:1428] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250904 07:53:38.310973 3507 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:38.313377 3507 tablet_bootstrap.cc:654] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.313933 3505 tablet_bootstrap.cc:654] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:38.316751 3506 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:38.317005 3506 ts_tablet_manager.cc:1397] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20250904 07:53:38.317948 3506 raft_consensus.cc:357] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.318228 3506 raft_consensus.cc:383] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.318356 3506 raft_consensus.cc:738] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.318661 3506 consensus_queue.cc:260] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.321539 3506 ts_tablet_manager.cc:1428] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250904 07:53:38.325258 3507 tablet_bootstrap.cc:492] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:38.325413 3507 ts_tablet_manager.cc:1397] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250904 07:53:38.326414 3505 tablet_bootstrap.cc:492] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:38.326535 3505 ts_tablet_manager.cc:1397] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20250904 07:53:38.326499 3507 raft_consensus.cc:357] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.326675 3507 raft_consensus.cc:383] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.326741 3507 raft_consensus.cc:738] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.326968 3507 consensus_queue.cc:260] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.327178 3505 raft_consensus.cc:357] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.327347 3505 raft_consensus.cc:383] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:38.327415 3505 raft_consensus.cc:738] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:38.327472 3507 ts_tablet_manager.cc:1428] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:38.327611 3505 consensus_queue.cc:260] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.328119 3505 ts_tablet_manager.cc:1428] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:38.329110 3437 raft_consensus.cc:491] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:38.329231 3437 raft_consensus.cc:513] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.329942 3437 leader_election.cc:290] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:38.330268 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2381750df7a49288423e2170a7c0838" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:38.330444 2290 raft_consensus.cc:2466] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:38.330833 2425 leader_election.cc:304] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:38.331423 3437 raft_consensus.cc:2802] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:38.331504 3437 raft_consensus.cc:491] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:38.331578 3437 raft_consensus.cc:3058] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.331666 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2381750df7a49288423e2170a7c0838" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:38.331802 2389 raft_consensus.cc:2466] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 0.
I20250904 07:53:38.333699 3437 raft_consensus.cc:513] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.334139 3437 leader_election.cc:290] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289)
I20250904 07:53:38.334441 2290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2381750df7a49288423e2170a7c0838" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:38.334503 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2381750df7a49288423e2170a7c0838" candidate_uuid: "7a00f455205e4b0eb7aba39391375837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:38.334594 2290 raft_consensus.cc:3058] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.334641 2389 raft_consensus.cc:3058] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.336242 2290 raft_consensus.cc:2466] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:38.336556 2425 leader_election.cc:304] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:38.336799 3437 raft_consensus.cc:2802] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:38.336941 3437 raft_consensus.cc:695] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Becoming Leader. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Running, Role: LEADER
I20250904 07:53:38.337077 2389 raft_consensus.cc:2466] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a00f455205e4b0eb7aba39391375837 in term 1.
I20250904 07:53:38.337173 3437 consensus_queue.cc:237] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:38.341146 2172 catalog_manager.cc:5582] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a00f455205e4b0eb7aba39391375837 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "7a00f455205e4b0eb7aba39391375837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:38.569876 3512 raft_consensus.cc:491] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:38.570050 3512 raft_consensus.cc:513] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.570667 3512 leader_election.cc:290] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:38.570881 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd732bc8933444cb35bd041088b2286" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:38.571064 2389 raft_consensus.cc:2466] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:38.571087 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd732bc8933444cb35bd041088b2286" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:38.571204 2464 raft_consensus.cc:2466] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:38.571435 2249 leader_election.cc:304] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:38.571774 3512 raft_consensus.cc:2802] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:38.571862 3512 raft_consensus.cc:491] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:38.571938 3512 raft_consensus.cc:3058] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.573719 3512 raft_consensus.cc:513] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.574167 3512 leader_election.cc:290] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:38.574600 2389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd732bc8933444cb35bd041088b2286" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:38.574602 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccd732bc8933444cb35bd041088b2286" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:38.574767 2389 raft_consensus.cc:3058] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.574813 2464 raft_consensus.cc:3058] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:38.576462 2389 raft_consensus.cc:2466] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:38.576510 2464 raft_consensus.cc:2466] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:38.576790 2249 leader_election.cc:304] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:38.577010 3464 raft_consensus.cc:2802] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:38.577121 3464 raft_consensus.cc:695] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:38.577353 3464 consensus_queue.cc:237] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:38.581168 2172 catalog_manager.cc:5582] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:38.641465 2387 raft_consensus.cc:1273] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:38.641861 2291 raft_consensus.cc:1273] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer 7a00f455205e4b0eb7aba39391375837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:38.641937 3499 consensus_queue.cc:1035] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:38.642356 2464 raft_consensus.cc:1273] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:38.642791 2387 raft_consensus.cc:1273] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:38.642812 3512 consensus_queue.cc:1035] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:38.643205 3499 consensus_queue.cc:1035] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:38.643656 3512 consensus_queue.cc:1035] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:38.970293 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 248ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:39.229074 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 509ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:39.486678 417 kernel_stack_watchdog.cc:198] Thread 2320 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 768ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:39.662405 2320 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.67:42657, user_credentials={real_user=slave}} blocked reactor thread for 1.00091e+06us
I20250904 07:53:39.663708 2170 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "table.31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\001\000\000\000""\004\001\000\001\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 07:53:39.664476 2170 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table.31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:53:39.676192 2379 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2d73adfb7046088a1eee593cf71635 (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:39.676290 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2d73adfb7046088a1eee593cf71635 (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:39.676615 2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d73adfb7046088a1eee593cf71635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.676658 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d73adfb7046088a1eee593cf71635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.676872 2378 tablet_service.cc:1468] Processing CreateTablet for tablet 5203d076bb4f4d2eacaa9c27ffd08adf (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:39.677184 2378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5203d076bb4f4d2eacaa9c27ffd08adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.682509 3536 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:39.682861 2278 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2d73adfb7046088a1eee593cf71635 (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION VALUES < 1
I20250904 07:53:39.683282 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2d73adfb7046088a1eee593cf71635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.683311 2280 tablet_service.cc:1468] Processing CreateTablet for tablet 5203d076bb4f4d2eacaa9c27ffd08adf (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:39.683689 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5203d076bb4f4d2eacaa9c27ffd08adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.684332 3536 tablet_bootstrap.cc:654] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.692559 3537 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:39.693112 2453 tablet_service.cc:1468] Processing CreateTablet for tablet 5203d076bb4f4d2eacaa9c27ffd08adf (DEFAULT_TABLE table=table.31 [id=eb8ca259fa7e4273900fe4cf486a08f1]), partition=RANGE (key) PARTITION 1 <= VALUES
I20250904 07:53:39.693800 2453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5203d076bb4f4d2eacaa9c27ffd08adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:39.694463 3537 tablet_bootstrap.cc:654] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.695947 3539 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:39.697919 3539 tablet_bootstrap.cc:654] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.711285 3537 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:39.711436 3537 ts_tablet_manager.cc:1397] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20250904 07:53:39.712041 3536 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:39.712213 3539 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:39.712251 3536 ts_tablet_manager.cc:1397] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.018s
I20250904 07:53:39.712394 3539 ts_tablet_manager.cc:1397] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20250904 07:53:39.712213 3537 raft_consensus.cc:357] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.712599 3537 raft_consensus.cc:383] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.712707 3537 raft_consensus.cc:738] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.712992 3537 consensus_queue.cc:260] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.713333 3536 raft_consensus.cc:357] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.713531 3536 raft_consensus.cc:383] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.713624 3536 raft_consensus.cc:738] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.713645 3537 ts_tablet_manager.cc:1428] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:39.713905 3536 consensus_queue.cc:260] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.714206 3537 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: Bootstrap starting.
I20250904 07:53:39.714428 3536 ts_tablet_manager.cc:1428] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:39.714885 3536 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: Bootstrap starting.
I20250904 07:53:39.714952 3539 raft_consensus.cc:357] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.715165 3539 raft_consensus.cc:383] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.715282 3539 raft_consensus.cc:738] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.715528 3539 consensus_queue.cc:260] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.716104 3537 tablet_bootstrap.cc:654] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.716576 3536 tablet_bootstrap.cc:654] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.718351 3539 ts_tablet_manager.cc:1428] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250904 07:53:39.718698 3539 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: Bootstrap starting.
I20250904 07:53:39.720937 3539 tablet_bootstrap.cc:654] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:39.724777 3536 tablet_bootstrap.cc:492] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: No bootstrap required, opened a new log
I20250904 07:53:39.724956 3536 ts_tablet_manager.cc:1397] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20250904 07:53:39.726089 3536 raft_consensus.cc:357] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.726284 3536 raft_consensus.cc:383] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.726356 3536 raft_consensus.cc:738] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a00f455205e4b0eb7aba39391375837, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.726598 3536 consensus_queue.cc:260] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.728528 3536 ts_tablet_manager.cc:1428] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:39.730484 3539 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: No bootstrap required, opened a new log
I20250904 07:53:39.730623 3539 ts_tablet_manager.cc:1397] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20250904 07:53:39.731302 3539 raft_consensus.cc:357] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.731454 3539 raft_consensus.cc:383] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.731503 3539 raft_consensus.cc:738] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.731667 3539 consensus_queue.cc:260] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.732188 3539 ts_tablet_manager.cc:1428] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:39.733521 3537 tablet_bootstrap.cc:492] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: No bootstrap required, opened a new log
I20250904 07:53:39.733685 3537 ts_tablet_manager.cc:1397] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20250904 07:53:39.734380 3537 raft_consensus.cc:357] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.734593 3537 raft_consensus.cc:383] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:39.734699 3537 raft_consensus.cc:738] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Initialized, Role: FOLLOWER
I20250904 07:53:39.734930 3537 consensus_queue.cc:260] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.735433 3537 ts_tablet_manager.cc:1428] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:39.925442 3500 raft_consensus.cc:491] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:39.925606 3500 raft_consensus.cc:513] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.926236 3500 leader_election.cc:290] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:39.926625 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5203d076bb4f4d2eacaa9c27ffd08adf" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:39.926803 2464 raft_consensus.cc:2466] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:39.927115 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5203d076bb4f4d2eacaa9c27ffd08adf" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106" is_pre_election: true
I20250904 07:53:39.927191 2345 leader_election.cc:304] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a00f455205e4b0eb7aba39391375837, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:39.927263 2291 raft_consensus.cc:2466] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 0.
I20250904 07:53:39.927424 3500 raft_consensus.cc:2802] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:39.927515 3500 raft_consensus.cc:491] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:39.927592 3500 raft_consensus.cc:3058] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:39.929503 3500 raft_consensus.cc:513] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.930068 3500 leader_election.cc:290] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Requested vote from peers 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:39.930346 2291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5203d076bb4f4d2eacaa9c27ffd08adf" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f35e2e598e4b4390f48a4b9888d106"
I20250904 07:53:39.930357 2464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5203d076bb4f4d2eacaa9c27ffd08adf" candidate_uuid: "a1047321e14f440c818a8f606e49cb81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:39.930477 2291 raft_consensus.cc:3058] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:39.930513 2464 raft_consensus.cc:3058] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:39.933084 2464 raft_consensus.cc:2466] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:39.933099 2291 raft_consensus.cc:2466] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1047321e14f440c818a8f606e49cb81 in term 1.
I20250904 07:53:39.933473 2345 leader_election.cc:304] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:39.933683 3500 raft_consensus.cc:2802] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:39.933804 3500 raft_consensus.cc:695] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Becoming Leader. State: Replica: a1047321e14f440c818a8f606e49cb81, State: Running, Role: LEADER
I20250904 07:53:39.934003 3500 consensus_queue.cc:237] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } }
I20250904 07:53:39.938560 2172 catalog_manager.cc:5582] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1047321e14f440c818a8f606e49cb81 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a1047321e14f440c818a8f606e49cb81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:39.994302 3531 raft_consensus.cc:491] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:39.994438 3531 raft_consensus.cc:513] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.994867 3531 leader_election.cc:290] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:39.995117 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2d73adfb7046088a1eee593cf71635" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81" is_pre_election: true
I20250904 07:53:39.995226 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2d73adfb7046088a1eee593cf71635" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837" is_pre_election: true
I20250904 07:53:39.995285 2388 raft_consensus.cc:2466] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:39.995383 2466 raft_consensus.cc:2466] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 0.
I20250904 07:53:39.995600 2249 leader_election.cc:304] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, a1047321e14f440c818a8f606e49cb81; no voters:
I20250904 07:53:39.995771 3531 raft_consensus.cc:2802] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:39.995841 3531 raft_consensus.cc:491] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:39.995909 3531 raft_consensus.cc:3058] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:39.998615 3531 raft_consensus.cc:513] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:39.999019 3531 leader_election.cc:290] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Requested vote from peers a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289), 7a00f455205e4b0eb7aba39391375837 (127.0.103.67:42657)
I20250904 07:53:39.999313 2388 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2d73adfb7046088a1eee593cf71635" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1047321e14f440c818a8f606e49cb81"
I20250904 07:53:39.999379 2466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2d73adfb7046088a1eee593cf71635" candidate_uuid: "52f35e2e598e4b4390f48a4b9888d106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a00f455205e4b0eb7aba39391375837"
I20250904 07:53:39.999450 2388 raft_consensus.cc:3058] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:39.999504 2466 raft_consensus.cc:3058] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:40.001798 2466 raft_consensus.cc:2466] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:40.001801 2388 raft_consensus.cc:2466] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f35e2e598e4b4390f48a4b9888d106 in term 1.
I20250904 07:53:40.002219 2247 leader_election.cc:304] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f35e2e598e4b4390f48a4b9888d106, 7a00f455205e4b0eb7aba39391375837; no voters:
I20250904 07:53:40.002447 3531 raft_consensus.cc:2802] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:40.002607 3531 raft_consensus.cc:695] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Becoming Leader. State: Replica: 52f35e2e598e4b4390f48a4b9888d106, State: Running, Role: LEADER
I20250904 07:53:40.002863 3531 consensus_queue.cc:237] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } }
I20250904 07:53:40.007187 2169 catalog_manager.cc:5582] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "52f35e2e598e4b4390f48a4b9888d106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:40.021631 2291 raft_consensus.cc:1273] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:40.021692 2466 raft_consensus.cc:1273] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer a1047321e14f440c818a8f606e49cb81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:40.022101 3500 consensus_queue.cc:1035] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f35e2e598e4b4390f48a4b9888d106" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:40.022231 2464 raft_consensus.cc:1273] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:40.022468 3500 consensus_queue.cc:1035] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:40.022727 2388 raft_consensus.cc:1273] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Refusing update from remote peer 52f35e2e598e4b4390f48a4b9888d106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:40.023175 3512 consensus_queue.cc:1035] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1047321e14f440c818a8f606e49cb81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:40.023512 3512 consensus_queue.cc:1035] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a00f455205e4b0eb7aba39391375837" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 07:53:40.345191 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 249ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:40.602578 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 506ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:40.862433 417 kernel_stack_watchdog.cc:198] Thread 2322 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 763ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fa1515f8980 (unknown) at ??:0
@ 0x7fa1515f7d50 __GI___nanosleep at ??:0
@ 0x7fa14d784e7b base::SleepForNanoseconds() at ??:0
@ 0x7fa14e890a46 kudu::SleepFor() at ??:0
@ 0x9bd884 kudu::client::ClientTest_TestAutoFlushBackgroundAndErrorCollector_Test::TestBody()::CustomErrorCollector::AddError() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:4458
@ 0x7fa15c461493 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0
@ 0x7fa15c45ff96 kudu::client::internal::WriteRpc::Finish() at ??:0
@ 0x7fa15c471375 kudu::rpc::RetriableRpc<>::SendRpcCb() at ??:0
@ 0x7fa15c48bc21 _ZZN4kudu3rpc12RetriableRpcINS_6client8internal18RemoteTabletServerENS_7tserver14WriteRequestPBENS5_15WriteResponsePBEE14ReplicaFoundCbERKNS_6StatusEPS4_ENKUlvE_clEv at ??:0
@ 0x7fa15c48b96e _ZNSt17_Function_handlerIFvvEZN4kudu3rpc12RetriableRpcINS1_6client8internal18RemoteTabletServerENS1_7tserver14WriteRequestPBENS7_15WriteResponsePBEE14ReplicaFoundCbERKNS1_6StatusEPS6_EUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xb2f51d std::function<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
@ 0x7fa155169860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fa155169dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fa1551052aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fa155103c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fa155116135 ev::base<>::method_thunk<>() at ??:0
W20250904 07:53:41.040195 2322 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.103.66:40289, user_credentials={real_user=slave}} blocked reactor thread for 1.00084e+06us
I20250904 07:53:41.041057 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:41.053944 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:41.054211 413 tablet_replica.cc:331] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.054433 413 raft_consensus.cc:2241] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.054817 413 raft_consensus.cc:2270] T 5422efe674664a778e97827177ba6482 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.055388 413 tablet_replica.cc:331] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.055562 413 raft_consensus.cc:2241] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.055720 413 raft_consensus.cc:2270] T 722589bedbd74a42a0918a9445743358 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.056154 413 tablet_replica.cc:331] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.056308 413 raft_consensus.cc:2241] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.056452 413 raft_consensus.cc:2270] T 2d07c349e0d2442085909737fa052037 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.056896 413 tablet_replica.cc:331] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.057039 413 raft_consensus.cc:2241] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.057188 413 raft_consensus.cc:2270] T 578c6279e6814dd49de7dae96142b3d5 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.057571 413 tablet_replica.cc:331] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.057720 413 raft_consensus.cc:2241] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.057857 413 raft_consensus.cc:2270] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.058301 413 tablet_replica.cc:331] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.058526 413 raft_consensus.cc:2241] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.058679 413 raft_consensus.cc:2270] T 956895d5d04b41169ec5c465cb0bc42a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.059156 413 tablet_replica.cc:331] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.059309 413 raft_consensus.cc:2241] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.059568 413 raft_consensus.cc:2270] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.060099 413 tablet_replica.cc:331] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
W20250904 07:53:41.060148 2425 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250904 07:53:41.060271 413 raft_consensus.cc:2241] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.060451 413 raft_consensus.cc:2270] T 15f3987a102544a180d8c8e0f34ac6bf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.061028 413 tablet_replica.cc:331] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.061220 413 raft_consensus.cc:2241] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
W20250904 07:53:41.061270 2425 consensus_peers.cc:489] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.061491 413 raft_consensus.cc:2270] T fd2f3a606d184b6dab7466c1bb848869 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.061894 413 tablet_replica.cc:331] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.062044 413 raft_consensus.cc:2241] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.062268 413 raft_consensus.cc:2270] T ccd732bc8933444cb35bd041088b2286 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.062783 413 tablet_replica.cc:331] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.062950 413 raft_consensus.cc:2241] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.063112 413 raft_consensus.cc:2270] T 4840a3a32d17474a83157fbf1853c7ce P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.063591 413 tablet_replica.cc:331] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.063735 413 raft_consensus.cc:2241] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.063982 413 raft_consensus.cc:2270] T 2a4522e61ea845658ce3694e9fef670c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.064476 413 tablet_replica.cc:331] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.064631 413 raft_consensus.cc:2241] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.064888 413 raft_consensus.cc:2270] T 479353cf35ca440aa4e7a9c70307a98f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.065351 413 tablet_replica.cc:331] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.065505 413 raft_consensus.cc:2241] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.065656 413 raft_consensus.cc:2270] T c2381750df7a49288423e2170a7c0838 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.066087 413 tablet_replica.cc:331] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.066251 413 raft_consensus.cc:2241] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.066493 413 raft_consensus.cc:2270] T f35cfd684d4f4cb1bfd92c5388b1f70c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.066975 413 tablet_replica.cc:331] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.067126 413 raft_consensus.cc:2241] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.067303 413 raft_consensus.cc:2270] T c3b8571e45544920a1425933d2bc0e33 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.067797 413 tablet_replica.cc:331] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.067962 413 raft_consensus.cc:2241] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.068146 413 raft_consensus.cc:2270] T 3335ab06a4b24194a43832dbc0ac5c51 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.068672 413 tablet_replica.cc:331] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.068837 413 raft_consensus.cc:2241] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.069128 413 raft_consensus.cc:2270] T 8f2ac889acc74693bd1816bdf8fe3d23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.069628 413 tablet_replica.cc:331] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.069772 413 raft_consensus.cc:2241] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.069938 413 raft_consensus.cc:2270] T f6efeb21885f46108d1c5a44471095b4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.070425 2345 consensus_peers.cc:489] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.070499 413 tablet_replica.cc:331] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.070647 413 raft_consensus.cc:2241] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.070914 413 raft_consensus.cc:2270] T 72304b06e03748b69f462d924a5ce342 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.071087 2425 consensus_peers.cc:489] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.071434 413 tablet_replica.cc:331] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.071595 413 raft_consensus.cc:2241] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.071808 413 raft_consensus.cc:2270] T 6a938a8ee4894764b610f1a154182a46 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.072291 413 tablet_replica.cc:331] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.072448 413 raft_consensus.cc:2241] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.072602 413 raft_consensus.cc:2270] T f633b3dc86a6495e859a188cecc671d0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.073196 413 tablet_replica.cc:331] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.073330 413 raft_consensus.cc:2241] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.073490 413 raft_consensus.cc:2270] T 821862116ac24420a0b4103bca1968e7 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.073908 2345 consensus_peers.cc:489] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.074008 413 tablet_replica.cc:331] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.074168 413 raft_consensus.cc:2241] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.074321 413 raft_consensus.cc:2270] T e2bd781edbe045e381aa09a6df6e4a1b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.074771 413 tablet_replica.cc:331] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.074930 413 raft_consensus.cc:2241] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.075150 413 raft_consensus.cc:2270] T 3d55b836537343439721fe076eff4ec0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.075555 413 tablet_replica.cc:331] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.075716 413 raft_consensus.cc:2241] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.075958 413 raft_consensus.cc:2270] T 422243e6ec0b46c68d96f46423bd8961 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.076455 413 tablet_replica.cc:331] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.076592 413 raft_consensus.cc:2241] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.076728 413 raft_consensus.cc:2270] T f4603a3c045a4016b557456a09da845c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.077204 413 tablet_replica.cc:331] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.077306 413 raft_consensus.cc:2241] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.077420 413 raft_consensus.cc:2270] T 527220ead94f46fe9dac566679a9c804 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.077898 413 tablet_replica.cc:331] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.078064 413 raft_consensus.cc:2241] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.078305 413 raft_consensus.cc:2270] T 1b6c13fce2c34de39f172711a420daab P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.078792 413 tablet_replica.cc:331] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.078944 413 raft_consensus.cc:2241] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.079216 413 raft_consensus.cc:2270] T 36e975e08e9542599e32bcc640a83b2a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.079604 413 tablet_replica.cc:331] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.079715 413 raft_consensus.cc:2241] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.079883 413 raft_consensus.cc:2270] T 8e123b0d29484c93abb544655194c12f P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.080271 413 tablet_replica.cc:331] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.080371 413 raft_consensus.cc:2241] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.080462 413 raft_consensus.cc:2270] T 2196fc4e937340e99876b54644e94905 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.080806 413 tablet_replica.cc:331] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.080953 413 raft_consensus.cc:2241] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.081138 413 raft_consensus.cc:2270] T cdd1e744f16b407980acc76a77789727 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.081480 413 tablet_replica.cc:331] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.081576 413 raft_consensus.cc:2241] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.081682 413 raft_consensus.cc:2270] T 3035b3b6d35d47d19186e2b55eb51012 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.082067 413 tablet_replica.cc:331] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.082162 413 raft_consensus.cc:2241] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.082274 413 raft_consensus.cc:2270] T 370e09c64ae047deb24be40c12777f0b P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.082610 413 tablet_replica.cc:331] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.082711 413 raft_consensus.cc:2241] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.082819 413 raft_consensus.cc:2270] T 64ee60c907814f64858eb85b6f555684 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.083164 413 tablet_replica.cc:331] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.083261 413 raft_consensus.cc:2241] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.083424 413 raft_consensus.cc:2270] T 56252ec8672745adaa3bf8af60e68f82 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.083757 413 tablet_replica.cc:331] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.083858 413 raft_consensus.cc:2241] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.083968 413 raft_consensus.cc:2270] T 523c68b1e547434cbc499d3c12e90a21 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.084367 413 tablet_replica.cc:331] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.084470 413 raft_consensus.cc:2241] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.084578 413 raft_consensus.cc:2270] T ce4fa00e235c4bca9188e975c48f9425 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.085043 413 tablet_replica.cc:331] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.085150 413 raft_consensus.cc:2241] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.085260 413 raft_consensus.cc:2270] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.085626 413 tablet_replica.cc:331] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.085762 413 raft_consensus.cc:2241] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.085901 413 raft_consensus.cc:2270] T 5203d076bb4f4d2eacaa9c27ffd08adf P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.086320 413 tablet_replica.cc:331] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.086455 413 raft_consensus.cc:2241] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.086591 413 raft_consensus.cc:2270] T 14bdb1ac7adf4767bafcc8964e63e885 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.086958 413 tablet_replica.cc:331] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.087095 413 raft_consensus.cc:2241] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.087231 413 raft_consensus.cc:2270] T cf534c94cabb4b49b5e6ee589c38501c P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.087610 413 tablet_replica.cc:331] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.087738 413 raft_consensus.cc:2241] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.087872 413 raft_consensus.cc:2270] T 1aa68c431d5c4e8a8d86d0295d024d5a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.088268 413 tablet_replica.cc:331] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.088397 413 raft_consensus.cc:2241] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.088531 413 raft_consensus.cc:2270] T eaa1415f6508476db015383017a473b0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.088939 413 tablet_replica.cc:331] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.089082 413 raft_consensus.cc:2241] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.089290 413 raft_consensus.cc:2270] T af9f6b7c4b144844ba8aa9efe6f9d483 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.089785 413 tablet_replica.cc:331] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.089927 413 raft_consensus.cc:2241] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.090092 413 raft_consensus.cc:2270] T ae6401dea08d46809de5df0d5358deba P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.090551 413 tablet_replica.cc:331] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.090711 413 raft_consensus.cc:2241] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.090975 413 raft_consensus.cc:2270] T afcd723aff4f478d96a0a8992d940916 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.091460 413 tablet_replica.cc:331] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.091598 413 raft_consensus.cc:2241] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.091785 413 raft_consensus.cc:2270] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.092275 413 tablet_replica.cc:331] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.092403 413 raft_consensus.cc:2241] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.092561 413 raft_consensus.cc:2270] T 5be5f30e8f684211b92a3e21486c6ed0 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.093066 413 tablet_replica.cc:331] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.093210 413 raft_consensus.cc:2241] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.093364 413 raft_consensus.cc:2270] T df55f6a7bfd84f01960d43ca38a627c3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.093746 413 tablet_replica.cc:331] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.093886 413 raft_consensus.cc:2241] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.094033 413 raft_consensus.cc:2270] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.094420 413 tablet_replica.cc:331] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.094558 413 raft_consensus.cc:2241] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.094702 413 raft_consensus.cc:2270] T b78c6741bce34db99dc3db5bf7db5ad6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.095103 413 tablet_replica.cc:331] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.095238 413 raft_consensus.cc:2241] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.095395 413 raft_consensus.cc:2270] T b5d23874cb8c4e58b959fcf93ca357f6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.095780 413 tablet_replica.cc:331] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.095913 413 raft_consensus.cc:2241] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.096101 413 raft_consensus.cc:2270] T 8f9c55548b294b4895483709aa0cd36d P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.096487 413 tablet_replica.cc:331] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.096621 413 raft_consensus.cc:2241] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.096762 413 raft_consensus.cc:2270] T 4911b93f2bd1424284b0517f51d64b65 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.097193 413 tablet_replica.cc:331] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.097328 413 raft_consensus.cc:2241] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.097541 413 raft_consensus.cc:2270] T e32aa65f41fe4672a9b2e2916e397a23 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.097929 413 tablet_replica.cc:331] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.098066 413 raft_consensus.cc:2241] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.098207 413 raft_consensus.cc:2270] T 4de16f56eb074e2a910185b0f63b403a P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.098588 413 tablet_replica.cc:331] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.098718 413 raft_consensus.cc:2241] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.098938 413 raft_consensus.cc:2270] T 0814b7aedf2f4e818d683ba3e0d000e8 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.099336 413 tablet_replica.cc:331] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.099467 413 raft_consensus.cc:2241] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.099606 413 raft_consensus.cc:2270] T 10a9f44c40154f53987c52fbd68e78d6 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.099977 413 tablet_replica.cc:331] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.100131 413 raft_consensus.cc:2241] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.100347 413 raft_consensus.cc:2270] T 6c2d73adfb7046088a1eee593cf71635 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.100754 413 tablet_replica.cc:331] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.100917 413 raft_consensus.cc:2241] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.101052 413 raft_consensus.cc:2270] T d207638da3bf43fc96e29e53cbd76fd4 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.101431 413 tablet_replica.cc:331] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.101554 413 raft_consensus.cc:2241] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.101692 413 raft_consensus.cc:2270] T 8c611852cad544fca62c3dae1e16cfa9 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.102078 413 tablet_replica.cc:331] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.102201 413 raft_consensus.cc:2241] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.102339 413 raft_consensus.cc:2270] T c75cefc8a85947cbbae7977ef83e3eff P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.102717 413 tablet_replica.cc:331] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.102846 413 raft_consensus.cc:2241] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.103003 413 raft_consensus.cc:2270] T 157bae761cee4869a60c2b508bab84c1 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.103467 413 tablet_replica.cc:331] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106: stopping tablet replica
I20250904 07:53:41.103585 413 raft_consensus.cc:2241] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.103726 413 raft_consensus.cc:2270] T 6dad7879f8424cb6a107dfe4e3671b29 P 52f35e2e598e4b4390f48a4b9888d106 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.107818 2345 consensus_peers.cc:489] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.110083 2345 consensus_peers.cc:489] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.118847 2248 connection.cc:441] server connection from 127.0.0.1:35488 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35488 (request call id 1314) could send its response
W20250904 07:53:41.119138 2345 consensus_peers.cc:489] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: recv got EOF from 127.0.103.65:46697 (error 108). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.121727 2425 consensus_peers.cc:489] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.128573 2425 consensus_peers.cc:489] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.131228 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
W20250904 07:53:41.133335 2345 consensus_peers.cc:489] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.135740 2345 consensus_peers.cc:489] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.138940 2425 consensus_peers.cc:489] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.145998 2425 consensus_peers.cc:489] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.156373 2425 consensus_peers.cc:489] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.165397 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20250904 07:53:41.166719 2427 consensus_peers.cc:489] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.168350 2427 consensus_peers.cc:489] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.176077 2345 consensus_peers.cc:489] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.204928 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:41.205245 413 tablet_replica.cc:331] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.205425 413 raft_consensus.cc:2241] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.205577 413 raft_consensus.cc:2270] T 5422efe674664a778e97827177ba6482 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.206050 413 tablet_replica.cc:331] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.206179 413 raft_consensus.cc:2241] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.206396 413 raft_consensus.cc:2270] T 722589bedbd74a42a0918a9445743358 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.206784 413 tablet_replica.cc:331] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.206907 413 raft_consensus.cc:2241] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.207089 413 raft_consensus.cc:2270] T 2d07c349e0d2442085909737fa052037 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.207459 413 tablet_replica.cc:331] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.207571 413 raft_consensus.cc:2241] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.207684 413 raft_consensus.cc:2270] T 578c6279e6814dd49de7dae96142b3d5 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.208052 413 tablet_replica.cc:331] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.208168 413 raft_consensus.cc:2241] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.208287 413 raft_consensus.cc:2270] T d8f84bd2ad024dbcb28d5f8befcb6c37 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.208635 413 tablet_replica.cc:331] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.208747 413 raft_consensus.cc:2241] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.208905 413 raft_consensus.cc:2270] T 956895d5d04b41169ec5c465cb0bc42a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.209293 413 tablet_replica.cc:331] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.209408 413 raft_consensus.cc:2241] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.209534 413 raft_consensus.cc:2270] T 0b8a122459d34c1494e5f8bf2a0c92b1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.209915 413 tablet_replica.cc:331] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.210026 413 raft_consensus.cc:2241] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.210141 413 raft_consensus.cc:2270] T 15f3987a102544a180d8c8e0f34ac6bf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.210491 413 tablet_replica.cc:331] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.210600 413 raft_consensus.cc:2241] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.210717 413 raft_consensus.cc:2270] T ccd732bc8933444cb35bd041088b2286 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.211068 413 tablet_replica.cc:331] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.211196 413 raft_consensus.cc:2241] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.211365 413 raft_consensus.cc:2270] T 4840a3a32d17474a83157fbf1853c7ce P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.211772 413 tablet_replica.cc:331] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.211938 413 raft_consensus.cc:2241] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.212085 413 raft_consensus.cc:2270] T fd2f3a606d184b6dab7466c1bb848869 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.212503 2345 consensus_peers.cc:489] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.212572 413 tablet_replica.cc:331] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.212711 413 raft_consensus.cc:2241] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.212879 413 raft_consensus.cc:2270] T 2a4522e61ea845658ce3694e9fef670c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.213317 413 tablet_replica.cc:331] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.213450 413 raft_consensus.cc:2241] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.213590 413 raft_consensus.cc:2270] T 479353cf35ca440aa4e7a9c70307a98f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.214006 413 tablet_replica.cc:331] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.214149 413 raft_consensus.cc:2241] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.214308 413 raft_consensus.cc:2270] T c2381750df7a49288423e2170a7c0838 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.214820 413 tablet_replica.cc:331] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.214974 413 raft_consensus.cc:2241] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.215129 413 raft_consensus.cc:2270] T f35cfd684d4f4cb1bfd92c5388b1f70c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.215673 2427 consensus_peers.cc:489] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.215673 413 tablet_replica.cc:331] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.215901 413 raft_consensus.cc:2241] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.216063 413 raft_consensus.cc:2270] T c3b8571e45544920a1425933d2bc0e33 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.216467 413 tablet_replica.cc:331] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.216602 413 raft_consensus.cc:2241] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.216826 413 raft_consensus.cc:2270] T 1aa68c431d5c4e8a8d86d0295d024d5a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.217273 413 tablet_replica.cc:331] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.217422 413 raft_consensus.cc:2241] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.217623 413 raft_consensus.cc:2270] T eaa1415f6508476db015383017a473b0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.218010 413 tablet_replica.cc:331] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.218144 413 raft_consensus.cc:2241] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.218284 413 raft_consensus.cc:2270] T f6efeb21885f46108d1c5a44471095b4 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.218676 413 tablet_replica.cc:331] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.218834 413 raft_consensus.cc:2241] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.218993 413 raft_consensus.cc:2270] T 72304b06e03748b69f462d924a5ce342 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.219430 413 tablet_replica.cc:331] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.219596 413 raft_consensus.cc:2241] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.219751 413 raft_consensus.cc:2270] T 6a938a8ee4894764b610f1a154182a46 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.220209 413 tablet_replica.cc:331] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.220374 413 raft_consensus.cc:2241] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.220623 413 raft_consensus.cc:2270] T 3335ab06a4b24194a43832dbc0ac5c51 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.220597 2345 consensus_peers.cc:489] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.221129 413 tablet_replica.cc:331] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.221246 413 raft_consensus.cc:2241] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.221416 413 raft_consensus.cc:2270] T 821862116ac24420a0b4103bca1968e7 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.221752 413 tablet_replica.cc:331] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.221858 413 raft_consensus.cc:2241] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.222028 413 raft_consensus.cc:2270] T e2bd781edbe045e381aa09a6df6e4a1b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.222380 413 tablet_replica.cc:331] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.222499 413 raft_consensus.cc:2241] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.222625 413 raft_consensus.cc:2270] T 3d55b836537343439721fe076eff4ec0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.222960 413 tablet_replica.cc:331] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.223064 413 raft_consensus.cc:2241] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.223173 413 raft_consensus.cc:2270] T 422243e6ec0b46c68d96f46423bd8961 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.223507 413 tablet_replica.cc:331] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.223603 413 raft_consensus.cc:2241] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.223766 413 raft_consensus.cc:2270] T f4603a3c045a4016b557456a09da845c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.224102 413 tablet_replica.cc:331] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.224200 413 raft_consensus.cc:2241] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.224370 413 raft_consensus.cc:2270] T f633b3dc86a6495e859a188cecc671d0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.224696 413 tablet_replica.cc:331] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.224798 413 raft_consensus.cc:2241] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.224969 413 raft_consensus.cc:2270] T 527220ead94f46fe9dac566679a9c804 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.225381 413 tablet_replica.cc:331] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.225521 413 raft_consensus.cc:2241] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.225667 413 raft_consensus.cc:2270] T 1b6c13fce2c34de39f172711a420daab P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.226120 413 tablet_replica.cc:331] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.226269 413 raft_consensus.cc:2241] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.226419 413 raft_consensus.cc:2270] T 36e975e08e9542599e32bcc640a83b2a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.226902 413 tablet_replica.cc:331] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
W20250904 07:53:41.226930 2425 consensus_peers.cc:489] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.227051 413 raft_consensus.cc:2241] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.227200 413 raft_consensus.cc:2270] T 8e123b0d29484c93abb544655194c12f P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.227595 413 tablet_replica.cc:331] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.227728 413 raft_consensus.cc:2241] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.227890 413 raft_consensus.cc:2270] T cdd1e744f16b407980acc76a77789727 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.228348 413 tablet_replica.cc:331] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.228480 413 raft_consensus.cc:2241] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.228688 413 raft_consensus.cc:2270] T 3035b3b6d35d47d19186e2b55eb51012 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.229135 413 tablet_replica.cc:331] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.229301 413 raft_consensus.cc:2241] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.229534 413 raft_consensus.cc:2270] T 370e09c64ae047deb24be40c12777f0b P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.229941 413 tablet_replica.cc:331] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.230077 413 raft_consensus.cc:2241] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.230284 413 raft_consensus.cc:2270] T 5203d076bb4f4d2eacaa9c27ffd08adf P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.230684 413 tablet_replica.cc:331] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.230813 413 raft_consensus.cc:2241] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.231006 413 raft_consensus.cc:2270] T 14bdb1ac7adf4767bafcc8964e63e885 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.231442 413 tablet_replica.cc:331] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.231582 413 raft_consensus.cc:2241] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.231724 413 raft_consensus.cc:2270] T af9f6b7c4b144844ba8aa9efe6f9d483 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.232118 413 tablet_replica.cc:331] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.232247 413 raft_consensus.cc:2241] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.232463 413 raft_consensus.cc:2270] T ae6401dea08d46809de5df0d5358deba P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.232896 413 tablet_replica.cc:331] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.233032 413 raft_consensus.cc:2241] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.233170 413 raft_consensus.cc:2270] T afcd723aff4f478d96a0a8992d940916 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.233572 413 tablet_replica.cc:331] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.233705 413 raft_consensus.cc:2241] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.233902 413 raft_consensus.cc:2270] T cf534c94cabb4b49b5e6ee589c38501c P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.234292 413 tablet_replica.cc:331] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.234416 413 raft_consensus.cc:2241] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.234617 413 raft_consensus.cc:2270] T 64ee60c907814f64858eb85b6f555684 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.235029 413 tablet_replica.cc:331] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.235155 413 raft_consensus.cc:2241] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.235299 413 raft_consensus.cc:2270] T 56252ec8672745adaa3bf8af60e68f82 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.235670 413 tablet_replica.cc:331] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.235800 413 raft_consensus.cc:2241] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.235932 413 raft_consensus.cc:2270] T 523c68b1e547434cbc499d3c12e90a21 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.236300 413 tablet_replica.cc:331] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.236418 413 raft_consensus.cc:2241] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.236608 413 raft_consensus.cc:2270] T ce4fa00e235c4bca9188e975c48f9425 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.237021 413 tablet_replica.cc:331] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.237166 413 raft_consensus.cc:2241] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.237310 413 raft_consensus.cc:2270] T 7d1766b3b4704b1fb8d60acf06c9c7eb P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.237675 413 tablet_replica.cc:331] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.237798 413 raft_consensus.cc:2241] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.237998 413 raft_consensus.cc:2270] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.238389 413 tablet_replica.cc:331] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.238516 413 raft_consensus.cc:2241] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.238648 413 raft_consensus.cc:2270] T 5be5f30e8f684211b92a3e21486c6ed0 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.239013 413 tablet_replica.cc:331] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.239176 413 raft_consensus.cc:2241] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.239373 413 raft_consensus.cc:2270] T df55f6a7bfd84f01960d43ca38a627c3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.239769 413 tablet_replica.cc:331] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.239905 413 raft_consensus.cc:2241] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.240051 413 raft_consensus.cc:2270] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.240492 413 tablet_replica.cc:331] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.240643 413 raft_consensus.cc:2241] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.240844 413 raft_consensus.cc:2270] T b78c6741bce34db99dc3db5bf7db5ad6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.241259 413 tablet_replica.cc:331] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.241345 413 raft_consensus.cc:2241] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.241441 413 raft_consensus.cc:2270] T 4911b93f2bd1424284b0517f51d64b65 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.241780 413 tablet_replica.cc:331] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.241907 413 raft_consensus.cc:2241] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.242121 413 raft_consensus.cc:2270] T 8f9c55548b294b4895483709aa0cd36d P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.242509 413 tablet_replica.cc:331] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.242643 413 raft_consensus.cc:2241] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.242841 413 raft_consensus.cc:2270] T b5d23874cb8c4e58b959fcf93ca357f6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.243216 413 tablet_replica.cc:331] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.243345 413 raft_consensus.cc:2241] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.243479 413 raft_consensus.cc:2270] T 8f2ac889acc74693bd1816bdf8fe3d23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.243835 413 tablet_replica.cc:331] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.243955 413 raft_consensus.cc:2241] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.244087 413 raft_consensus.cc:2270] T 0814b7aedf2f4e818d683ba3e0d000e8 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.244458 413 tablet_replica.cc:331] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.244580 413 raft_consensus.cc:2241] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.244712 413 raft_consensus.cc:2270] T 10a9f44c40154f53987c52fbd68e78d6 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.245111 413 tablet_replica.cc:331] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.245275 413 raft_consensus.cc:2241] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.245412 413 raft_consensus.cc:2270] T e32aa65f41fe4672a9b2e2916e397a23 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.245812 413 tablet_replica.cc:331] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.245932 413 raft_consensus.cc:2241] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.246069 413 raft_consensus.cc:2270] T 4de16f56eb074e2a910185b0f63b403a P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.246454 413 tablet_replica.cc:331] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.246573 413 raft_consensus.cc:2241] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.246709 413 raft_consensus.cc:2270] T 6c2d73adfb7046088a1eee593cf71635 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.247074 413 tablet_replica.cc:331] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.247192 413 raft_consensus.cc:2241] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.247334 413 raft_consensus.cc:2270] T 8c611852cad544fca62c3dae1e16cfa9 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.247695 413 tablet_replica.cc:331] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.247828 413 raft_consensus.cc:2241] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.248044 413 raft_consensus.cc:2270] T c75cefc8a85947cbbae7977ef83e3eff P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.248427 413 tablet_replica.cc:331] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.248546 413 raft_consensus.cc:2241] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.248744 413 raft_consensus.cc:2270] T 157bae761cee4869a60c2b508bab84c1 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.249159 413 tablet_replica.cc:331] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81: stopping tablet replica
I20250904 07:53:41.249279 413 raft_consensus.cc:2241] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.249416 413 raft_consensus.cc:2270] T 6dad7879f8424cb6a107dfe4e3671b29 P a1047321e14f440c818a8f606e49cb81 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.270169 2427 consensus_peers.cc:489] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.273455 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
W20250904 07:53:41.285336 2427 consensus_peers.cc:489] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.302484 2427 consensus_peers.cc:489] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250904 07:53:41.304468 2427 consensus_peers.cc:489] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.308821 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
W20250904 07:53:41.319898 2425 consensus_peers.cc:489] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.320420 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:41.320639 413 tablet_replica.cc:331] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.320820 413 raft_consensus.cc:2241] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.321025 413 raft_consensus.cc:2270] T 5422efe674664a778e97827177ba6482 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.321537 413 tablet_replica.cc:331] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.321686 413 raft_consensus.cc:2241] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.321827 413 raft_consensus.cc:2270] T 722589bedbd74a42a0918a9445743358 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.322252 413 tablet_replica.cc:331] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.322386 413 raft_consensus.cc:2241] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.322608 413 raft_consensus.cc:2270] T 578c6279e6814dd49de7dae96142b3d5 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.323007 413 tablet_replica.cc:331] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.323189 413 raft_consensus.cc:2241] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.323345 413 raft_consensus.cc:2270] T 2d07c349e0d2442085909737fa052037 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.323724 413 tablet_replica.cc:331] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.323863 413 raft_consensus.cc:2241] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.324079 413 raft_consensus.cc:2270] T d8f84bd2ad024dbcb28d5f8befcb6c37 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.324467 413 tablet_replica.cc:331] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.324602 413 raft_consensus.cc:2241] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.324810 413 raft_consensus.cc:2270] T 956895d5d04b41169ec5c465cb0bc42a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.325222 413 tablet_replica.cc:331] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.325358 413 raft_consensus.cc:2241] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.325518 413 raft_consensus.cc:2270] T 0b8a122459d34c1494e5f8bf2a0c92b1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.326005 413 tablet_replica.cc:331] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.326159 413 raft_consensus.cc:2241] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.326385 413 raft_consensus.cc:2270] T 15f3987a102544a180d8c8e0f34ac6bf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.326891 413 tablet_replica.cc:331] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
W20250904 07:53:41.327028 2425 consensus_peers.cc:489] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.327047 413 raft_consensus.cc:2241] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.327236 413 raft_consensus.cc:2270] T fd2f3a606d184b6dab7466c1bb848869 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.327629 413 tablet_replica.cc:331] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.327766 413 raft_consensus.cc:2241] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.327952 413 raft_consensus.cc:2270] T ccd732bc8933444cb35bd041088b2286 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.328424 413 tablet_replica.cc:331] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.328581 413 raft_consensus.cc:2241] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.328776 413 raft_consensus.cc:2270] T 4840a3a32d17474a83157fbf1853c7ce P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250904 07:53:41.329234 2427 consensus_peers.cc:489] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 -> Peer a1047321e14f440c818a8f606e49cb81 (127.0.103.66:40289): Couldn't send request to peer a1047321e14f440c818a8f606e49cb81. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.329321 413 tablet_replica.cc:331] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.329471 413 raft_consensus.cc:2241] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.329617 413 raft_consensus.cc:2270] T 2a4522e61ea845658ce3694e9fef670c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.330061 413 tablet_replica.cc:331] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.330193 413 raft_consensus.cc:2241] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.330356 413 raft_consensus.cc:2270] T 479353cf35ca440aa4e7a9c70307a98f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.330753 413 tablet_replica.cc:331] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.330888 413 raft_consensus.cc:2241] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.331099 413 raft_consensus.cc:2270] T c2381750df7a49288423e2170a7c0838 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.331478 413 tablet_replica.cc:331] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.331610 413 raft_consensus.cc:2241] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.331748 413 raft_consensus.cc:2270] T f35cfd684d4f4cb1bfd92c5388b1f70c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.332140 413 tablet_replica.cc:331] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.332273 413 raft_consensus.cc:2241] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.332489 413 raft_consensus.cc:2270] T c3b8571e45544920a1425933d2bc0e33 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.332891 413 tablet_replica.cc:331] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.333031 413 raft_consensus.cc:2241] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.333168 413 raft_consensus.cc:2270] T 1aa68c431d5c4e8a8d86d0295d024d5a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.333539 413 tablet_replica.cc:331] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.333700 413 raft_consensus.cc:2241] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.333835 413 raft_consensus.cc:2270] T eaa1415f6508476db015383017a473b0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.334215 413 tablet_replica.cc:331] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.334343 413 raft_consensus.cc:2241] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.334542 413 raft_consensus.cc:2270] T f6efeb21885f46108d1c5a44471095b4 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.334929 413 tablet_replica.cc:331] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.335067 413 raft_consensus.cc:2241] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.335278 413 raft_consensus.cc:2270] T 72304b06e03748b69f462d924a5ce342 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.335640 413 tablet_replica.cc:331] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.335779 413 raft_consensus.cc:2241] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.335913 413 raft_consensus.cc:2270] T 6a938a8ee4894764b610f1a154182a46 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.336316 413 tablet_replica.cc:331] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.336450 413 raft_consensus.cc:2241] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.336587 413 raft_consensus.cc:2270] T 3335ab06a4b24194a43832dbc0ac5c51 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.337000 413 tablet_replica.cc:331] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.337133 413 raft_consensus.cc:2241] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.337283 413 raft_consensus.cc:2270] T f633b3dc86a6495e859a188cecc671d0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.337670 413 tablet_replica.cc:331] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.337815 413 raft_consensus.cc:2241] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.337946 413 raft_consensus.cc:2270] T 821862116ac24420a0b4103bca1968e7 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.338308 413 tablet_replica.cc:331] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.338430 413 raft_consensus.cc:2241] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.338562 413 raft_consensus.cc:2270] T e2bd781edbe045e381aa09a6df6e4a1b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.338943 413 tablet_replica.cc:331] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.339075 413 raft_consensus.cc:2241] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.339217 413 raft_consensus.cc:2270] T 3d55b836537343439721fe076eff4ec0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.339588 413 tablet_replica.cc:331] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.339717 413 raft_consensus.cc:2241] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.339855 413 raft_consensus.cc:2270] T 422243e6ec0b46c68d96f46423bd8961 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.340226 413 tablet_replica.cc:331] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.340350 413 raft_consensus.cc:2241] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.340484 413 raft_consensus.cc:2270] T f4603a3c045a4016b557456a09da845c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.340847 413 tablet_replica.cc:331] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.341055 413 raft_consensus.cc:2241] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.341274 413 raft_consensus.cc:2270] T 527220ead94f46fe9dac566679a9c804 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.341670 413 tablet_replica.cc:331] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.341801 413 raft_consensus.cc:2241] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.341949 413 raft_consensus.cc:2270] T 1b6c13fce2c34de39f172711a420daab P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.342346 413 tablet_replica.cc:331] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.342475 413 raft_consensus.cc:2241] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.342614 413 raft_consensus.cc:2270] T 36e975e08e9542599e32bcc640a83b2a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.342994 413 tablet_replica.cc:331] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.343149 413 raft_consensus.cc:2241] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.343310 413 raft_consensus.cc:2270] T 8e123b0d29484c93abb544655194c12f P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.343698 413 tablet_replica.cc:331] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.343827 413 raft_consensus.cc:2241] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.343966 413 raft_consensus.cc:2270] T cdd1e744f16b407980acc76a77789727 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.344353 413 tablet_replica.cc:331] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.344491 413 raft_consensus.cc:2241] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.344642 413 raft_consensus.cc:2270] T 3035b3b6d35d47d19186e2b55eb51012 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.345134 413 tablet_replica.cc:331] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.345296 413 raft_consensus.cc:2241] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.345463 413 raft_consensus.cc:2270] T 370e09c64ae047deb24be40c12777f0b P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.345963 413 tablet_replica.cc:331] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
W20250904 07:53:41.345974 2425 consensus_peers.cc:489] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.346132 413 raft_consensus.cc:2241] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.346268 413 raft_consensus.cc:2270] T af9f6b7c4b144844ba8aa9efe6f9d483 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.346616 413 tablet_replica.cc:331] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.346716 413 raft_consensus.cc:2241] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.346834 413 raft_consensus.cc:2270] T ae6401dea08d46809de5df0d5358deba P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.347177 413 tablet_replica.cc:331] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.347290 413 raft_consensus.cc:2241] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.347396 413 raft_consensus.cc:2270] T afcd723aff4f478d96a0a8992d940916 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.347724 413 tablet_replica.cc:331] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.347823 413 raft_consensus.cc:2241] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.347944 413 raft_consensus.cc:2270] T cf534c94cabb4b49b5e6ee589c38501c P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.348297 413 tablet_replica.cc:331] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.348405 413 raft_consensus.cc:2241] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.348513 413 raft_consensus.cc:2270] T 5203d076bb4f4d2eacaa9c27ffd08adf P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.348850 413 tablet_replica.cc:331] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.349001 413 raft_consensus.cc:2241] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.349109 413 raft_consensus.cc:2270] T 14bdb1ac7adf4767bafcc8964e63e885 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.349457 413 tablet_replica.cc:331] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.349571 413 raft_consensus.cc:2241] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.349736 413 raft_consensus.cc:2270] T 64ee60c907814f64858eb85b6f555684 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.350217 413 tablet_replica.cc:331] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.350347 413 raft_consensus.cc:2241] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.350502 413 raft_consensus.cc:2270] T 56252ec8672745adaa3bf8af60e68f82 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.350893 413 tablet_replica.cc:331] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.351033 413 raft_consensus.cc:2241] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
W20250904 07:53:41.351073 2425 consensus_peers.cc:489] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 -> Peer 52f35e2e598e4b4390f48a4b9888d106 (127.0.103.65:46697): Couldn't send request to peer 52f35e2e598e4b4390f48a4b9888d106. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:41.351279 413 raft_consensus.cc:2270] T 523c68b1e547434cbc499d3c12e90a21 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.351661 413 tablet_replica.cc:331] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.351789 413 raft_consensus.cc:2241] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.351923 413 raft_consensus.cc:2270] T ce4fa00e235c4bca9188e975c48f9425 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.352296 413 tablet_replica.cc:331] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.352434 413 raft_consensus.cc:2241] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.352627 413 raft_consensus.cc:2270] T 7d1766b3b4704b1fb8d60acf06c9c7eb P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.353055 413 tablet_replica.cc:331] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.353179 413 raft_consensus.cc:2241] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.353338 413 raft_consensus.cc:2270] T c016ef01d8eb4df4a8cf9fbcbf7ba1e0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.353724 413 tablet_replica.cc:331] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.353865 413 raft_consensus.cc:2241] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.354063 413 raft_consensus.cc:2270] T 5be5f30e8f684211b92a3e21486c6ed0 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.354434 413 tablet_replica.cc:331] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.354549 413 raft_consensus.cc:2241] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.354672 413 raft_consensus.cc:2270] T df55f6a7bfd84f01960d43ca38a627c3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.355024 413 tablet_replica.cc:331] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.355134 413 raft_consensus.cc:2241] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.355340 413 raft_consensus.cc:2270] T 0fa8abb8eb304fe7be4ade8a1afbd9e3 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.355690 413 tablet_replica.cc:331] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.355796 413 raft_consensus.cc:2241] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.355918 413 raft_consensus.cc:2270] T b78c6741bce34db99dc3db5bf7db5ad6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.356272 413 tablet_replica.cc:331] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.356380 413 raft_consensus.cc:2241] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.356524 413 raft_consensus.cc:2270] T b5d23874cb8c4e58b959fcf93ca357f6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.356891 413 tablet_replica.cc:331] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.357015 413 raft_consensus.cc:2241] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.357138 413 raft_consensus.cc:2270] T 8f2ac889acc74693bd1816bdf8fe3d23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.357520 413 tablet_replica.cc:331] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.357628 413 raft_consensus.cc:2241] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.357750 413 raft_consensus.cc:2270] T 8f9c55548b294b4895483709aa0cd36d P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.358100 413 tablet_replica.cc:331] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.358215 413 raft_consensus.cc:2241] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.358410 413 raft_consensus.cc:2270] T 4911b93f2bd1424284b0517f51d64b65 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.358772 413 tablet_replica.cc:331] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.358883 413 raft_consensus.cc:2241] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.359012 413 raft_consensus.cc:2270] T 0814b7aedf2f4e818d683ba3e0d000e8 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.359385 413 tablet_replica.cc:331] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.359494 413 raft_consensus.cc:2241] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.359691 413 raft_consensus.cc:2270] T 10a9f44c40154f53987c52fbd68e78d6 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.360061 413 tablet_replica.cc:331] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.360167 413 raft_consensus.cc:2241] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.360302 413 raft_consensus.cc:2270] T e32aa65f41fe4672a9b2e2916e397a23 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.360656 413 tablet_replica.cc:331] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.360762 413 raft_consensus.cc:2241] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.361009 413 raft_consensus.cc:2270] T 4de16f56eb074e2a910185b0f63b403a P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.361400 413 tablet_replica.cc:331] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.361513 413 raft_consensus.cc:2241] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.361642 413 raft_consensus.cc:2270] T 6c2d73adfb7046088a1eee593cf71635 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.362023 413 tablet_replica.cc:331] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.362131 413 raft_consensus.cc:2241] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.362334 413 raft_consensus.cc:2270] T 8c611852cad544fca62c3dae1e16cfa9 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.362701 413 tablet_replica.cc:331] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.362807 413 raft_consensus.cc:2241] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.362946 413 raft_consensus.cc:2270] T c75cefc8a85947cbbae7977ef83e3eff P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.363327 413 tablet_replica.cc:331] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.363430 413 raft_consensus.cc:2241] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:41.363554 413 raft_consensus.cc:2270] T 157bae761cee4869a60c2b508bab84c1 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.363946 413 tablet_replica.cc:331] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837: stopping tablet replica
I20250904 07:53:41.364050 413 raft_consensus.cc:2241] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.364244 413 raft_consensus.cc:2270] T 6dad7879f8424cb6a107dfe4e3671b29 P 7a00f455205e4b0eb7aba39391375837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.396134 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250904 07:53:41.429560 413 master.cc:561] Master@127.0.103.126:38915 shutting down...
I20250904 07:53:41.438730 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.438941 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.439047 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7f4f1af1682c402d8305da8c5d5f1da7: stopping tablet replica
I20250904 07:53:41.484072 413 master.cc:583] Master@127.0.103.126:38915 shutdown complete.
[ OK ] ClientTest.TestAutoFlushBackgroundAndErrorCollector (43420 ms)
[ RUN ] ClientTest.TestMutationsWork
I20250904 07:53:41.539774 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:44043
I20250904 07:53:41.540282 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:41.542786 3566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.542760 3567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.542963 3569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:41.543453 413 server_base.cc:1047] running on GCE node
I20250904 07:53:41.543713 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:41.543761 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:41.543800 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972421543797 us; error 0 us; skew 500 ppm
I20250904 07:53:41.543946 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:41.545243 413 webserver.cc:480] Webserver started at http://127.0.103.126:35157/ using document root <none> and password file <none>
I20250904 07:53:41.545433 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:41.545488 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:41.545600 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:41.546172 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "76f8a07e37f54bff828a457b5ca11e27"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.548360 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:41.549854 3574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.550233 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:41.550323 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "76f8a07e37f54bff828a457b5ca11e27"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.550401 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:41.560998 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:41.561383 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:41.573174 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44043
I20250904 07:53:41.573191 3635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44043 every 8 connection(s)
I20250904 07:53:41.574316 3636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.578030 3636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27: Bootstrap starting.
I20250904 07:53:41.579219 3636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.580605 3636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27: No bootstrap required, opened a new log
I20250904 07:53:41.581180 3636 raft_consensus.cc:357] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER }
I20250904 07:53:41.581310 3636 raft_consensus.cc:383] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.581362 3636 raft_consensus.cc:738] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f8a07e37f54bff828a457b5ca11e27, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.581555 3636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER }
I20250904 07:53:41.581653 3636 raft_consensus.cc:397] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.581710 3636 raft_consensus.cc:491] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.581792 3636 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.583375 3636 raft_consensus.cc:513] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER }
I20250904 07:53:41.583541 3636 leader_election.cc:304] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f8a07e37f54bff828a457b5ca11e27; no voters:
I20250904 07:53:41.583890 3636 leader_election.cc:290] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.584105 3639 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.584592 3639 raft_consensus.cc:695] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 1 LEADER]: Becoming Leader. State: Replica: 76f8a07e37f54bff828a457b5ca11e27, State: Running, Role: LEADER
I20250904 07:53:41.584618 3636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:41.584807 3639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER }
I20250904 07:53:41.585839 3641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f8a07e37f54bff828a457b5ca11e27. Latest consensus state: current_term: 1 leader_uuid: "76f8a07e37f54bff828a457b5ca11e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER } }
I20250904 07:53:41.585999 3641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:41.586743 3640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f8a07e37f54bff828a457b5ca11e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f8a07e37f54bff828a457b5ca11e27" member_type: VOTER } }
I20250904 07:53:41.586854 3640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:41.587184 3646 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:41.588433 3646 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:41.589841 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:41.591025 3646 catalog_manager.cc:1349] Generated new cluster ID: 56c94433436a44fd944ec57feea08cdb
I20250904 07:53:41.591123 3646 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:41.602540 3646 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:41.603260 3646 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:41.610558 3646 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27: Generated new TSK 0
I20250904 07:53:41.610750 3646 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:41.622229 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:41.624464 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.624759 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.625016 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:41.625717 413 server_base.cc:1047] running on GCE node
I20250904 07:53:41.625927 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:41.625986 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:41.626029 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972421626025 us; error 0 us; skew 500 ppm
I20250904 07:53:41.626158 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:41.627189 413 webserver.cc:480] Webserver started at http://127.0.103.65:46269/ using document root <none> and password file <none>
I20250904 07:53:41.627370 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:41.627431 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:41.627543 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:41.628093 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "67612f6c9fca46b6afa427db350390ee"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.630327 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:41.631709 3665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.632113 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:41.632198 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "67612f6c9fca46b6afa427db350390ee"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.632277 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestMutationsWork.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:41.643724 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:41.644114 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:41.645619 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:41.645701 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.645802 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:41.645871 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.661589 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39133
I20250904 07:53:41.661612 3735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39133 every 8 connection(s)
I20250904 07:53:41.663269 3736 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44043
I20250904 07:53:41.663396 3736 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:41.663712 3736 heartbeater.cc:507] Master 127.0.103.126:44043 requested a full tablet report, sending...
I20250904 07:53:41.664546 3591 ts_manager.cc:194] Registered new tserver with Master: 67612f6c9fca46b6afa427db350390ee (127.0.103.65:39133)
I20250904 07:53:41.664868 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002299007s
I20250904 07:53:41.665688 3591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59506
I20250904 07:53:41.674854 3591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59532:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:41.676823 3736 heartbeater.cc:499] Master 127.0.103.126:44043 was elected leader, sending a full tablet report...
I20250904 07:53:41.687094 3695 tablet_service.cc:1468] Processing CreateTablet for tablet f4b7d27bb0a64f2484c6d2a4c135f2c3 (DEFAULT_TABLE table=client-testtb [id=f37355eab6ac4dcc862feeba4e391c21]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:41.687094 3696 tablet_service.cc:1468] Processing CreateTablet for tablet 3b756651c42545b18f887b0ba4379071 (DEFAULT_TABLE table=client-testtb [id=f37355eab6ac4dcc862feeba4e391c21]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:41.687616 3695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b7d27bb0a64f2484c6d2a4c135f2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.687830 3696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b756651c42545b18f887b0ba4379071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.694109 3746 tablet_bootstrap.cc:492] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: Bootstrap starting.
I20250904 07:53:41.695751 3746 tablet_bootstrap.cc:654] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.698050 3746 tablet_bootstrap.cc:492] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: No bootstrap required, opened a new log
I20250904 07:53:41.698177 3746 ts_tablet_manager.cc:1397] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:41.698863 3746 raft_consensus.cc:357] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.698994 3746 raft_consensus.cc:383] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.699050 3746 raft_consensus.cc:738] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67612f6c9fca46b6afa427db350390ee, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.699240 3746 consensus_queue.cc:260] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.699386 3746 raft_consensus.cc:397] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.699445 3746 raft_consensus.cc:491] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.699514 3746 raft_consensus.cc:3058] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.701606 3746 raft_consensus.cc:513] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.701784 3746 leader_election.cc:304] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67612f6c9fca46b6afa427db350390ee; no voters:
I20250904 07:53:41.702132 3746 leader_election.cc:290] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.702234 3748 raft_consensus.cc:2802] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.702646 3748 raft_consensus.cc:695] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 1 LEADER]: Becoming Leader. State: Replica: 67612f6c9fca46b6afa427db350390ee, State: Running, Role: LEADER
I20250904 07:53:41.702817 3746 ts_tablet_manager.cc:1428] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250904 07:53:41.702842 3748 consensus_queue.cc:237] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.703059 3746 tablet_bootstrap.cc:492] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: Bootstrap starting.
I20250904 07:53:41.704599 3746 tablet_bootstrap.cc:654] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.704730 3591 catalog_manager.cc:5582] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 67612f6c9fca46b6afa427db350390ee (127.0.103.65). New cstate: current_term: 1 leader_uuid: "67612f6c9fca46b6afa427db350390ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:41.706125 3746 tablet_bootstrap.cc:492] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: No bootstrap required, opened a new log
I20250904 07:53:41.706241 3746 ts_tablet_manager.cc:1397] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:41.706859 3746 raft_consensus.cc:357] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.706990 3746 raft_consensus.cc:383] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.707048 3746 raft_consensus.cc:738] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67612f6c9fca46b6afa427db350390ee, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.707242 3746 consensus_queue.cc:260] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.707371 3746 raft_consensus.cc:397] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.707440 3746 raft_consensus.cc:491] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.707515 3746 raft_consensus.cc:3058] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.709812 3746 raft_consensus.cc:513] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.709952 3746 leader_election.cc:304] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67612f6c9fca46b6afa427db350390ee; no voters:
I20250904 07:53:41.710060 3746 leader_election.cc:290] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.710129 3748 raft_consensus.cc:2802] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.710264 3748 raft_consensus.cc:695] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 1 LEADER]: Becoming Leader. State: Replica: 67612f6c9fca46b6afa427db350390ee, State: Running, Role: LEADER
I20250904 07:53:41.710357 3746 ts_tablet_manager.cc:1428] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:41.710446 3748 consensus_queue.cc:237] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } }
I20250904 07:53:41.712049 3591 catalog_manager.cc:5582] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 67612f6c9fca46b6afa427db350390ee (127.0.103.65). New cstate: current_term: 1 leader_uuid: "67612f6c9fca46b6afa427db350390ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67612f6c9fca46b6afa427db350390ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39133 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:41.747817 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:41.773020 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:41.773343 413 tablet_replica.cc:331] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee: stopping tablet replica
I20250904 07:53:41.773519 413 raft_consensus.cc:2241] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.773686 413 raft_consensus.cc:2270] T f4b7d27bb0a64f2484c6d2a4c135f2c3 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.774428 413 tablet_replica.cc:331] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee: stopping tablet replica
I20250904 07:53:41.774559 413 raft_consensus.cc:2241] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.774708 413 raft_consensus.cc:2270] T 3b756651c42545b18f887b0ba4379071 P 67612f6c9fca46b6afa427db350390ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.789009 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:41.792980 413 master.cc:561] Master@127.0.103.126:44043 shutting down...
I20250904 07:53:41.800956 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:41.801133 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:41.801270 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 76f8a07e37f54bff828a457b5ca11e27: stopping tablet replica
I20250904 07:53:41.814694 413 master.cc:583] Master@127.0.103.126:44043 shutdown complete.
[ OK ] ClientTest.TestMutationsWork (283 ms)
[ RUN ] ClientTest.TestSoftDeleteAndReserveTable
I20250904 07:53:41.823364 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:44555
I20250904 07:53:41.823771 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:41.825592 3757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.825726 3758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.826156 3760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:41.826637 413 server_base.cc:1047] running on GCE node
I20250904 07:53:41.826869 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:41.826917 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:41.826939 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972421826936 us; error 0 us; skew 500 ppm
I20250904 07:53:41.827039 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:41.828034 413 webserver.cc:480] Webserver started at http://127.0.103.126:41667/ using document root <none> and password file <none>
I20250904 07:53:41.828183 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:41.828235 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:41.828312 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:41.828796 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "c9413c2d1cb442e088a8014689326892"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.830937 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:41.832278 3765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.832572 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:41.832669 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "c9413c2d1cb442e088a8014689326892"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.832759 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:41.839623 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:41.840018 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:41.853024 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44555
I20250904 07:53:41.853052 3826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44555 every 8 connection(s)
I20250904 07:53:41.854106 3827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.858053 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: Bootstrap starting.
I20250904 07:53:41.859342 3827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.860785 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: No bootstrap required, opened a new log
I20250904 07:53:41.861379 3827 raft_consensus.cc:357] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER }
I20250904 07:53:41.861491 3827 raft_consensus.cc:383] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.861544 3827 raft_consensus.cc:738] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9413c2d1cb442e088a8014689326892, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.861716 3827 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER }
I20250904 07:53:41.861815 3827 raft_consensus.cc:397] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.861886 3827 raft_consensus.cc:491] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.861966 3827 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.863643 3827 raft_consensus.cc:513] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER }
I20250904 07:53:41.863812 3827 leader_election.cc:304] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9413c2d1cb442e088a8014689326892; no voters:
I20250904 07:53:41.864154 3827 leader_election.cc:290] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.864297 3830 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.864778 3830 raft_consensus.cc:695] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 1 LEADER]: Becoming Leader. State: Replica: c9413c2d1cb442e088a8014689326892, State: Running, Role: LEADER
I20250904 07:53:41.865029 3830 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER }
I20250904 07:53:41.865126 3827 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:41.865646 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9413c2d1cb442e088a8014689326892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER } }
I20250904 07:53:41.865677 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9413c2d1cb442e088a8014689326892. Latest consensus state: current_term: 1 leader_uuid: "c9413c2d1cb442e088a8014689326892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9413c2d1cb442e088a8014689326892" member_type: VOTER } }
I20250904 07:53:41.865796 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:41.865805 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:41.866364 3834 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:41.867973 3834 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:41.870399 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:41.870492 3834 catalog_manager.cc:1349] Generated new cluster ID: f7113a7bc0d8432e97d0d2816d8a1f68
I20250904 07:53:41.870560 3834 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:41.877514 3834 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:41.878187 3834 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:41.891371 3834 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: Generated new TSK 0
I20250904 07:53:41.891553 3834 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:41.902844 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:41.905287 3848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.905478 3849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:41.905974 3851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:41.906116 413 server_base.cc:1047] running on GCE node
I20250904 07:53:41.906477 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:41.906543 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:41.906580 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972421906576 us; error 0 us; skew 500 ppm
I20250904 07:53:41.906701 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:41.907691 413 webserver.cc:480] Webserver started at http://127.0.103.65:39433/ using document root <none> and password file <none>
I20250904 07:53:41.907884 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:41.907979 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:41.908092 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:41.908632 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "5066134a8a4f4018890c5d05d2a87cea"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.910830 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:41.912226 3856 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.912513 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:41.912616 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "5066134a8a4f4018890c5d05d2a87cea"
format_stamp: "Formatted at 2025-09-04 07:53:41 on dist-test-slave-ghlz"
I20250904 07:53:41.912707 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:41.924326 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:41.924727 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:41.926219 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:41.926296 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.926378 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:41.926430 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:41.944974 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:37633
I20250904 07:53:41.944991 3926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:37633 every 8 connection(s)
I20250904 07:53:41.946252 3927 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44555
I20250904 07:53:41.946350 3927 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:41.946635 3927 heartbeater.cc:507] Master 127.0.103.126:44555 requested a full tablet report, sending...
I20250904 07:53:41.947310 3782 ts_manager.cc:194] Registered new tserver with Master: 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65:37633)
I20250904 07:53:41.948045 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002257672s
I20250904 07:53:41.948195 3782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55044
I20250904 07:53:41.956933 3782 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55076:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:41.959323 3927 heartbeater.cc:499] Master 127.0.103.126:44555 was elected leader, sending a full tablet report...
I20250904 07:53:41.968539 3889 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb5c3a92e8c428f8f3c3971c2c05ad8 (DEFAULT_TABLE table=client-testtb [id=5e3b73c246974e85963ac572d28742a7]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:41.968561 3887 tablet_service.cc:1468] Processing CreateTablet for tablet b471515c65504c578bcb30eb406ec43e (DEFAULT_TABLE table=client-testtb [id=5e3b73c246974e85963ac572d28742a7]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:41.969017 3889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb5c3a92e8c428f8f3c3971c2c05ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.969264 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b471515c65504c578bcb30eb406ec43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:41.973855 3937 tablet_bootstrap.cc:492] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Bootstrap starting.
I20250904 07:53:41.975044 3937 tablet_bootstrap.cc:654] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.976503 3937 tablet_bootstrap.cc:492] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: No bootstrap required, opened a new log
I20250904 07:53:41.976630 3937 ts_tablet_manager.cc:1397] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:41.977366 3937 raft_consensus.cc:357] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.977489 3937 raft_consensus.cc:383] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.977551 3937 raft_consensus.cc:738] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5066134a8a4f4018890c5d05d2a87cea, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.977736 3937 consensus_queue.cc:260] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.977847 3937 raft_consensus.cc:397] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.977909 3937 raft_consensus.cc:491] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.977998 3937 raft_consensus.cc:3058] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.979539 3937 raft_consensus.cc:513] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.979722 3937 leader_election.cc:304] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5066134a8a4f4018890c5d05d2a87cea; no voters:
I20250904 07:53:41.980131 3937 leader_election.cc:290] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.980235 3939 raft_consensus.cc:2802] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.980736 3939 raft_consensus.cc:695] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 1 LEADER]: Becoming Leader. State: Replica: 5066134a8a4f4018890c5d05d2a87cea, State: Running, Role: LEADER
I20250904 07:53:41.980986 3939 consensus_queue.cc:237] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.981204 3937 ts_tablet_manager.cc:1428] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250904 07:53:41.981515 3937 tablet_bootstrap.cc:492] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Bootstrap starting.
I20250904 07:53:41.982851 3782 catalog_manager.cc:5582] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5066134a8a4f4018890c5d05d2a87cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:41.983412 3937 tablet_bootstrap.cc:654] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:41.985306 3937 tablet_bootstrap.cc:492] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: No bootstrap required, opened a new log
I20250904 07:53:41.985436 3937 ts_tablet_manager.cc:1397] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:41.986078 3937 raft_consensus.cc:357] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.986249 3937 raft_consensus.cc:383] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:41.986332 3937 raft_consensus.cc:738] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5066134a8a4f4018890c5d05d2a87cea, State: Initialized, Role: FOLLOWER
I20250904 07:53:41.986526 3937 consensus_queue.cc:260] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.986692 3937 raft_consensus.cc:397] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:41.986753 3937 raft_consensus.cc:491] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:41.986835 3937 raft_consensus.cc:3058] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:41.988910 3937 raft_consensus.cc:513] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.989100 3937 leader_election.cc:304] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5066134a8a4f4018890c5d05d2a87cea; no voters:
I20250904 07:53:41.989245 3937 leader_election.cc:290] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:41.989295 3939 raft_consensus.cc:2802] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:41.989461 3939 raft_consensus.cc:695] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 1 LEADER]: Becoming Leader. State: Replica: 5066134a8a4f4018890c5d05d2a87cea, State: Running, Role: LEADER
I20250904 07:53:41.989600 3937 ts_tablet_manager.cc:1428] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:41.989670 3939 consensus_queue.cc:237] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } }
I20250904 07:53:41.991252 3781 catalog_manager.cc:5582] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5066134a8a4f4018890c5d05d2a87cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5066134a8a4f4018890c5d05d2a87cea" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37633 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.014635 3782 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } modify_external_catalogs: true reserve_seconds: 600
I20250904 07:53:42.017524 3782 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20250904 07:53:42.022166 3948 tablet.cc:1722] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250904 07:53:42.022249 3949 tablet.cc:1722] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250904 07:53:42.025765 3782 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } new_table_name: "client-testtb" modify_external_catalogs: true
I20250904 07:53:42.027395 3782 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "int_val" } } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "new_col" type: INT32 is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20250904 07:53:42.028541 3782 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } modify_external_catalogs: true reserve_seconds: 600
I20250904 07:53:42.029233 3781 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: client-testtb [id=5e3b73c246974e85963ac572d28742a7] alter complete (version 1)
I20250904 07:53:42.029788 3781 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } modify_external_catalogs: true new_extra_configs { key: "kudu.table.maintenance_priority" value: "3" }
I20250904 07:53:42.060613 3781 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55076:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:42.062434 3781 catalog_manager.cc:2869] Servicing RecallDeletedTableRpc request from {username='slave'} at 127.0.0.1:55076:
table { table_id: "5e3b73c246974e85963ac572d28742a7" }
I20250904 07:53:42.065255 3781 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55076:
table { table_id: "5e3b73c246974e85963ac572d28742a7" } new_extra_configs { key: "kudu.table.disable_compaction" value: "false" }
I20250904 07:53:42.069779 3948 tablet.cc:1722] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250904 07:53:42.070123 3949 tablet.cc:1722] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Alter schema from (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250904 07:53:42.071305 3782 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } modify_external_catalogs: true reserve_seconds: 0
I20250904 07:53:42.071473 3782 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55076:
table { table_name: "client-testtb" } modify_external_catalogs: true reserve_seconds: 0
I20250904 07:53:42.076050 3782 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: Sending DeleteTablet for 1 replicas of tablet 9bb5c3a92e8c428f8f3c3971c2c05ad8
I20250904 07:53:42.076277 3782 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: Sending DeleteTablet for 1 replicas of tablet b471515c65504c578bcb30eb406ec43e
I20250904 07:53:42.076570 3887 tablet_service.cc:1515] Processing DeleteTablet for tablet b471515c65504c578bcb30eb406ec43e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-04 07:53:42 UTC) from {username='slave'} at 127.0.0.1:35840
I20250904 07:53:42.076875 3889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bb5c3a92e8c428f8f3c3971c2c05ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-04 07:53:42 UTC) from {username='slave'} at 127.0.0.1:35840
I20250904 07:53:42.076912 3885 tablet_service.cc:1515] Processing DeleteTablet for tablet b471515c65504c578bcb30eb406ec43e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-04 07:53:42 UTC) from {username='slave'} at 127.0.0.1:35840
I20250904 07:53:42.077168 3950 tablet_replica.cc:331] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: stopping tablet replica
I20250904 07:53:42.077332 3950 raft_consensus.cc:2241] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.077549 3950 raft_consensus.cc:2270] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.078507 3950 ts_tablet_manager.cc:1905] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 07:53:42.082496 3950 ts_tablet_manager.cc:1918] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250904 07:53:42.082576 3950 log.cc:1199] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Deleting WAL directory at /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root/wals/b471515c65504c578bcb30eb406ec43e
I20250904 07:53:42.082850 3950 ts_tablet_manager.cc:1939] T b471515c65504c578bcb30eb406ec43e P 5066134a8a4f4018890c5d05d2a87cea: Deleting consensus metadata
I20250904 07:53:42.083817 3950 tablet_replica.cc:331] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: stopping tablet replica
I20250904 07:53:42.083912 3766 catalog_manager.cc:4928] TS 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65:37633): tablet b471515c65504c578bcb30eb406ec43e (table client-testtb [id=5e3b73c246974e85963ac572d28742a7]) successfully deleted
I20250904 07:53:42.083981 3950 raft_consensus.cc:2241] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.084143 3950 raft_consensus.cc:2270] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.084550 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:42.085992 3950 ts_tablet_manager.cc:1905] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 07:53:42.093614 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:42.094542 3950 ts_tablet_manager.cc:1918] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250904 07:53:42.094633 3950 log.cc:1199] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Deleting WAL directory at /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestSoftDeleteAndReserveTable.1756972373064810-413-0/minicluster-data/ts-0-root/wals/9bb5c3a92e8c428f8f3c3971c2c05ad8
I20250904 07:53:42.094960 3950 ts_tablet_manager.cc:1939] T 9bb5c3a92e8c428f8f3c3971c2c05ad8 P 5066134a8a4f4018890c5d05d2a87cea: Deleting consensus metadata
W20250904 07:53:42.095528 3766 catalog_manager.cc:4920] TS 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65:37633): delete failed for tablet 9bb5c3a92e8c428f8f3c3971c2c05ad8 with error code TABLET_NOT_RUNNING: Service unavailable: Tablet Manager is not running: MANAGER_QUIESCING
W20250904 07:53:42.096637 3766 catalog_manager.cc:4671] TS 5066134a8a4f4018890c5d05d2a87cea (127.0.103.65:37633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b471515c65504c578bcb30eb406ec43e: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20250904 07:53:42.109668 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:42.114256 413 master.cc:561] Master@127.0.103.126:44555 shutting down...
I20250904 07:53:42.158584 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.158847 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.159035 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P c9413c2d1cb442e088a8014689326892: stopping tablet replica
I20250904 07:53:42.162868 413 master.cc:583] Master@127.0.103.126:44555 shutdown complete.
[ OK ] ClientTest.TestSoftDeleteAndReserveTable (347 ms)
[ RUN ] ClientTest.TestGetTableSchema
I20250904 07:53:42.171406 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46015
I20250904 07:53:42.171850 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.173846 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.173972 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.174573 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.174930 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.175194 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.175247 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.175269 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422175267 us; error 0 us; skew 500 ppm
I20250904 07:53:42.175402 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.176393 413 webserver.cc:480] Webserver started at http://127.0.103.126:44795/ using document root <none> and password file <none>
I20250904 07:53:42.176565 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.176620 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.176728 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.177378 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "6132c5b2aca64d6588ba14227cdf4a80"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.179481 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:42.180739 3960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.181044 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:42.181136 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "6132c5b2aca64d6588ba14227cdf4a80"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.181223 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.194443 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.194854 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.207026 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46015
I20250904 07:53:42.207041 4021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46015 every 8 connection(s)
I20250904 07:53:42.208127 4022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.211835 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80: Bootstrap starting.
I20250904 07:53:42.213052 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.214428 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80: No bootstrap required, opened a new log
I20250904 07:53:42.214941 4022 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER }
I20250904 07:53:42.215050 4022 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.215082 4022 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6132c5b2aca64d6588ba14227cdf4a80, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.215247 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER }
I20250904 07:53:42.215350 4022 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.215389 4022 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.215437 4022 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.216987 4022 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER }
I20250904 07:53:42.217144 4022 leader_election.cc:304] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6132c5b2aca64d6588ba14227cdf4a80; no voters:
I20250904 07:53:42.217453 4022 leader_election.cc:290] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.217557 4025 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.217958 4025 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6132c5b2aca64d6588ba14227cdf4a80, State: Running, Role: LEADER
I20250904 07:53:42.218178 4025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER }
I20250904 07:53:42.218333 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:42.219103 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6132c5b2aca64d6588ba14227cdf4a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER } }
I20250904 07:53:42.219175 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6132c5b2aca64d6588ba14227cdf4a80. Latest consensus state: current_term: 1 leader_uuid: "6132c5b2aca64d6588ba14227cdf4a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6132c5b2aca64d6588ba14227cdf4a80" member_type: VOTER } }
I20250904 07:53:42.219399 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.220113 4031 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:42.220096 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.221560 4031 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:42.223393 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:42.224220 4031 catalog_manager.cc:1349] Generated new cluster ID: e44d0ca6cd24485ba54110c5e2464c3f
I20250904 07:53:42.224293 4031 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:42.239941 4031 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:42.240624 4031 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:42.245620 4031 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80: Generated new TSK 0
I20250904 07:53:42.245821 4031 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:42.256088 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.258471 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.258517 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.258862 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.260001 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.260257 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.260329 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.260392 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422260371 us; error 0 us; skew 500 ppm
I20250904 07:53:42.260545 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.261669 413 webserver.cc:480] Webserver started at http://127.0.103.65:38171/ using document root <none> and password file <none>
I20250904 07:53:42.261869 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.261951 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.262094 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.262639 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "bc273af65acd410ba8630f31e1479482"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.264744 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:53:42.266160 4051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.266446 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:42.266561 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "bc273af65acd410ba8630f31e1479482"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.266661 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestGetTableSchema.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.280741 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.281248 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.282824 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:42.283177 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.283460 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:42.283553 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.301515 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35925
I20250904 07:53:42.301544 4121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:35925 every 8 connection(s)
I20250904 07:53:42.302843 4122 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46015
I20250904 07:53:42.302985 4122 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:42.303282 4122 heartbeater.cc:507] Master 127.0.103.126:46015 requested a full tablet report, sending...
I20250904 07:53:42.303993 3977 ts_manager.cc:194] Registered new tserver with Master: bc273af65acd410ba8630f31e1479482 (127.0.103.65:35925)
I20250904 07:53:42.304658 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002323311s
I20250904 07:53:42.304946 3977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35306
I20250904 07:53:42.314057 3977 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35328:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:42.316011 4122 heartbeater.cc:499] Master 127.0.103.126:46015 was elected leader, sending a full tablet report...
I20250904 07:53:42.325939 4086 tablet_service.cc:1468] Processing CreateTablet for tablet cbed98165f21453a9a2b4a73dd246e73 (DEFAULT_TABLE table=client-testtb [id=2e8110b5e1a8421bbbe799df7d795ede]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:42.325955 4087 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba548889bb749d1a7aea7b3ba4311bc (DEFAULT_TABLE table=client-testtb [id=2e8110b5e1a8421bbbe799df7d795ede]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:42.326359 4086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbed98165f21453a9a2b4a73dd246e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.326588 4087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba548889bb749d1a7aea7b3ba4311bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.331199 4132 tablet_bootstrap.cc:492] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: Bootstrap starting.
I20250904 07:53:42.332410 4132 tablet_bootstrap.cc:654] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.333957 4132 tablet_bootstrap.cc:492] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: No bootstrap required, opened a new log
I20250904 07:53:42.334087 4132 ts_tablet_manager.cc:1397] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:42.334621 4132 raft_consensus.cc:357] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.334743 4132 raft_consensus.cc:383] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.334795 4132 raft_consensus.cc:738] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc273af65acd410ba8630f31e1479482, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.334944 4132 consensus_queue.cc:260] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.335045 4132 raft_consensus.cc:397] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.335099 4132 raft_consensus.cc:491] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.335160 4132 raft_consensus.cc:3058] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.336596 4132 raft_consensus.cc:513] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.336755 4132 leader_election.cc:304] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc273af65acd410ba8630f31e1479482; no voters:
I20250904 07:53:42.337105 4132 leader_election.cc:290] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.337235 4134 raft_consensus.cc:2802] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.337877 4134 raft_consensus.cc:695] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 1 LEADER]: Becoming Leader. State: Replica: bc273af65acd410ba8630f31e1479482, State: Running, Role: LEADER
I20250904 07:53:42.337971 4132 ts_tablet_manager.cc:1428] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250904 07:53:42.338137 4134 consensus_queue.cc:237] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.338251 4132 tablet_bootstrap.cc:492] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: Bootstrap starting.
I20250904 07:53:42.339879 4132 tablet_bootstrap.cc:654] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.340102 3977 catalog_manager.cc:5582] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc273af65acd410ba8630f31e1479482 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "bc273af65acd410ba8630f31e1479482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.341444 4132 tablet_bootstrap.cc:492] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: No bootstrap required, opened a new log
I20250904 07:53:42.341571 4132 ts_tablet_manager.cc:1397] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:42.342214 4132 raft_consensus.cc:357] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.342340 4132 raft_consensus.cc:383] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.342423 4132 raft_consensus.cc:738] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc273af65acd410ba8630f31e1479482, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.342615 4132 consensus_queue.cc:260] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.342747 4132 raft_consensus.cc:397] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.342829 4132 raft_consensus.cc:491] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.366106 4132 raft_consensus.cc:3058] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.368134 4132 raft_consensus.cc:513] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.368350 4132 leader_election.cc:304] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc273af65acd410ba8630f31e1479482; no voters:
I20250904 07:53:42.368535 4132 leader_election.cc:290] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.368633 4134 raft_consensus.cc:2802] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.368757 4134 raft_consensus.cc:695] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 1 LEADER]: Becoming Leader. State: Replica: bc273af65acd410ba8630f31e1479482, State: Running, Role: LEADER
I20250904 07:53:42.368940 4132 ts_tablet_manager.cc:1428] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: Time spent starting tablet: real 0.027s user 0.028s sys 0.000s
I20250904 07:53:42.369009 4134 consensus_queue.cc:237] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } }
I20250904 07:53:42.370663 3977 catalog_manager.cc:5582] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc273af65acd410ba8630f31e1479482 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "bc273af65acd410ba8630f31e1479482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc273af65acd410ba8630f31e1479482" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35925 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.386113 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:42.393828 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:42.394106 413 tablet_replica.cc:331] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482: stopping tablet replica
I20250904 07:53:42.394331 413 raft_consensus.cc:2241] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.394481 413 raft_consensus.cc:2270] T 4ba548889bb749d1a7aea7b3ba4311bc P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.395176 413 tablet_replica.cc:331] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482: stopping tablet replica
I20250904 07:53:42.395273 413 raft_consensus.cc:2241] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.395373 413 raft_consensus.cc:2270] T cbed98165f21453a9a2b4a73dd246e73 P bc273af65acd410ba8630f31e1479482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.399494 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:42.403453 413 master.cc:561] Master@127.0.103.126:46015 shutting down...
I20250904 07:53:42.411463 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.411665 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.411814 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6132c5b2aca64d6588ba14227cdf4a80: stopping tablet replica
I20250904 07:53:42.425626 413 master.cc:583] Master@127.0.103.126:46015 shutdown complete.
[ OK ] ClientTest.TestGetTableSchema (263 ms)
[ RUN ] ClientTest.TestCreateDuplicateTable
I20250904 07:53:42.434757 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:32885
I20250904 07:53:42.435163 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.437063 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.437204 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.437573 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.438041 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.438280 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.438333 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.438364 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422438359 us; error 0 us; skew 500 ppm
I20250904 07:53:42.438493 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.439469 413 webserver.cc:480] Webserver started at http://127.0.103.126:33083/ using document root <none> and password file <none>
I20250904 07:53:42.439648 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.439707 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.439816 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.440367 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "f273bd8eb37f420a9608287a348c0741"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.442802 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:42.444159 4149 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.444489 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:42.444573 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "f273bd8eb37f420a9608287a348c0741"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.444651 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.457577 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.457965 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.470968 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:32885
I20250904 07:53:42.470994 4210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:32885 every 8 connection(s)
I20250904 07:53:42.472105 4211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.475983 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741: Bootstrap starting.
I20250904 07:53:42.477243 4211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.478669 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741: No bootstrap required, opened a new log
I20250904 07:53:42.479264 4211 raft_consensus.cc:357] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER }
I20250904 07:53:42.479383 4211 raft_consensus.cc:383] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.479439 4211 raft_consensus.cc:738] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f273bd8eb37f420a9608287a348c0741, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.479611 4211 consensus_queue.cc:260] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER }
I20250904 07:53:42.479712 4211 raft_consensus.cc:397] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.479770 4211 raft_consensus.cc:491] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.479838 4211 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.481587 4211 raft_consensus.cc:513] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER }
I20250904 07:53:42.481753 4211 leader_election.cc:304] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f273bd8eb37f420a9608287a348c0741; no voters:
I20250904 07:53:42.482131 4211 leader_election.cc:290] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.482206 4214 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.482614 4214 raft_consensus.cc:695] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 1 LEADER]: Becoming Leader. State: Replica: f273bd8eb37f420a9608287a348c0741, State: Running, Role: LEADER
I20250904 07:53:42.482808 4214 consensus_queue.cc:237] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER }
I20250904 07:53:42.482857 4211 sys_catalog.cc:564] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:42.483410 4215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f273bd8eb37f420a9608287a348c0741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER } }
I20250904 07:53:42.483449 4216 sys_catalog.cc:455] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f273bd8eb37f420a9608287a348c0741. Latest consensus state: current_term: 1 leader_uuid: "f273bd8eb37f420a9608287a348c0741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f273bd8eb37f420a9608287a348c0741" member_type: VOTER } }
I20250904 07:53:42.483577 4215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.483585 4216 sys_catalog.cc:458] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.484836 4219 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:42.485978 4219 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:42.489138 4219 catalog_manager.cc:1349] Generated new cluster ID: 5fe3bb505bf647d8bdfdfd59d4e7b1b5
I20250904 07:53:42.489228 4219 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:42.490864 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:42.496819 4219 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:42.497529 4219 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:42.506251 4219 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741: Generated new TSK 0
I20250904 07:53:42.506414 4219 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:42.523363 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.525574 4232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.525601 4233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.526522 4235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.526957 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.527199 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.527249 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.527276 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422527274 us; error 0 us; skew 500 ppm
I20250904 07:53:42.527367 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.528390 413 webserver.cc:480] Webserver started at http://127.0.103.65:38259/ using document root <none> and password file <none>
I20250904 07:53:42.528546 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.528592 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.528689 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.529259 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "649c9597e11b4c0399816aef362c45aa"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.531298 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:42.532630 4240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.532971 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:42.533080 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "649c9597e11b4c0399816aef362c45aa"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.533193 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestCreateDuplicateTable.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.550061 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.550513 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.551931 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:42.552026 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.552134 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:42.552222 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.569060 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:43157
I20250904 07:53:42.569109 4310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:43157 every 8 connection(s)
I20250904 07:53:42.570375 4311 heartbeater.cc:344] Connected to a master server at 127.0.103.126:32885
I20250904 07:53:42.570485 4311 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:42.570726 4311 heartbeater.cc:507] Master 127.0.103.126:32885 requested a full tablet report, sending...
I20250904 07:53:42.571394 4166 ts_manager.cc:194] Registered new tserver with Master: 649c9597e11b4c0399816aef362c45aa (127.0.103.65:43157)
I20250904 07:53:42.572140 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002254453s
I20250904 07:53:42.572335 4166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37998
I20250904 07:53:42.582067 4166 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38020:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:42.583422 4311 heartbeater.cc:499] Master 127.0.103.126:32885 was elected leader, sending a full tablet report...
I20250904 07:53:42.593973 4272 tablet_service.cc:1468] Processing CreateTablet for tablet 6349af1ac8234a6884c514090bf2d87a (DEFAULT_TABLE table=client-testtb [id=7204784d48bc4795b889bb9821fc13a9]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:42.594025 4270 tablet_service.cc:1468] Processing CreateTablet for tablet 7fff4aea9377430a946eaf3d349ac810 (DEFAULT_TABLE table=client-testtb [id=7204784d48bc4795b889bb9821fc13a9]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:42.594431 4272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6349af1ac8234a6884c514090bf2d87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.594658 4270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fff4aea9377430a946eaf3d349ac810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.600478 4321 tablet_bootstrap.cc:492] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: Bootstrap starting.
I20250904 07:53:42.601888 4321 tablet_bootstrap.cc:654] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.603461 4321 tablet_bootstrap.cc:492] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: No bootstrap required, opened a new log
I20250904 07:53:42.603566 4321 ts_tablet_manager.cc:1397] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:42.604075 4321 raft_consensus.cc:357] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.604199 4321 raft_consensus.cc:383] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.604254 4321 raft_consensus.cc:738] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649c9597e11b4c0399816aef362c45aa, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.604400 4321 consensus_queue.cc:260] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.604504 4321 raft_consensus.cc:397] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.604555 4321 raft_consensus.cc:491] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.604616 4321 raft_consensus.cc:3058] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.606305 4321 raft_consensus.cc:513] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.606550 4321 leader_election.cc:304] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649c9597e11b4c0399816aef362c45aa; no voters:
I20250904 07:53:42.607317 4321 leader_election.cc:290] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.607405 4323 raft_consensus.cc:2802] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.607801 4323 raft_consensus.cc:695] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 1 LEADER]: Becoming Leader. State: Replica: 649c9597e11b4c0399816aef362c45aa, State: Running, Role: LEADER
I20250904 07:53:42.607993 4321 ts_tablet_manager.cc:1428] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250904 07:53:42.608075 4323 consensus_queue.cc:237] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.608246 4321 tablet_bootstrap.cc:492] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: Bootstrap starting.
I20250904 07:53:42.610061 4321 tablet_bootstrap.cc:654] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.610615 4166 catalog_manager.cc:5582] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 649c9597e11b4c0399816aef362c45aa (127.0.103.65). New cstate: current_term: 1 leader_uuid: "649c9597e11b4c0399816aef362c45aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.611852 4321 tablet_bootstrap.cc:492] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: No bootstrap required, opened a new log
I20250904 07:53:42.611984 4321 ts_tablet_manager.cc:1397] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:53:42.612733 4321 raft_consensus.cc:357] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.612912 4321 raft_consensus.cc:383] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.613005 4321 raft_consensus.cc:738] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649c9597e11b4c0399816aef362c45aa, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.613210 4321 consensus_queue.cc:260] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.613358 4321 raft_consensus.cc:397] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.613443 4321 raft_consensus.cc:491] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.613567 4321 raft_consensus.cc:3058] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.615370 4321 raft_consensus.cc:513] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.615554 4321 leader_election.cc:304] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649c9597e11b4c0399816aef362c45aa; no voters:
I20250904 07:53:42.615726 4321 leader_election.cc:290] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.615796 4324 raft_consensus.cc:2802] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.616164 4321 ts_tablet_manager.cc:1428] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:42.616298 4324 raft_consensus.cc:695] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 1 LEADER]: Becoming Leader. State: Replica: 649c9597e11b4c0399816aef362c45aa, State: Running, Role: LEADER
I20250904 07:53:42.616495 4324 consensus_queue.cc:237] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } }
I20250904 07:53:42.619438 4166 catalog_manager.cc:5582] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 649c9597e11b4c0399816aef362c45aa (127.0.103.65). New cstate: current_term: 1 leader_uuid: "649c9597e11b4c0399816aef362c45aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649c9597e11b4c0399816aef362c45aa" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43157 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.630327 4166 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38020:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:42.632371 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:42.640136 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:42.640486 413 tablet_replica.cc:331] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa: stopping tablet replica
I20250904 07:53:42.640748 413 raft_consensus.cc:2241] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.641036 413 raft_consensus.cc:2270] T 6349af1ac8234a6884c514090bf2d87a P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.642115 413 tablet_replica.cc:331] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa: stopping tablet replica
I20250904 07:53:42.642277 413 raft_consensus.cc:2241] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.642429 413 raft_consensus.cc:2270] T 7fff4aea9377430a946eaf3d349ac810 P 649c9597e11b4c0399816aef362c45aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.656947 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:42.660815 413 master.cc:561] Master@127.0.103.126:32885 shutting down...
I20250904 07:53:42.669420 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:42.669624 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:42.669766 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P f273bd8eb37f420a9608287a348c0741: stopping tablet replica
I20250904 07:53:42.674540 413 master.cc:583] Master@127.0.103.126:32885 shutdown complete.
[ OK ] ClientTest.TestCreateDuplicateTable (248 ms)
[ RUN ] ClientTest.TestInsertEmptyPK
I20250904 07:53:42.683674 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35647
I20250904 07:53:42.684131 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.686324 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.686558 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.686894 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.687208 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.687549 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.687603 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.687639 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422687635 us; error 0 us; skew 500 ppm
I20250904 07:53:42.687778 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.688968 413 webserver.cc:480] Webserver started at http://127.0.103.126:43727/ using document root <none> and password file <none>
I20250904 07:53:42.689170 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.689234 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.689347 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.689917 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "ceb8e073375545dfb45f34e0560280db"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.692200 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:53:42.693797 4339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.694152 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:42.694239 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "ceb8e073375545dfb45f34e0560280db"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.694322 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.710209 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.710638 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.723031 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35647
I20250904 07:53:42.723052 4400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35647 every 8 connection(s)
I20250904 07:53:42.724107 4401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.727880 4401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db: Bootstrap starting.
I20250904 07:53:42.729184 4401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.730643 4401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db: No bootstrap required, opened a new log
I20250904 07:53:42.731205 4401 raft_consensus.cc:357] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER }
I20250904 07:53:42.731313 4401 raft_consensus.cc:383] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.731364 4401 raft_consensus.cc:738] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb8e073375545dfb45f34e0560280db, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.731546 4401 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER }
I20250904 07:53:42.731645 4401 raft_consensus.cc:397] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.731705 4401 raft_consensus.cc:491] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.731776 4401 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.733425 4401 raft_consensus.cc:513] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER }
I20250904 07:53:42.733567 4401 leader_election.cc:304] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb8e073375545dfb45f34e0560280db; no voters:
I20250904 07:53:42.733911 4401 leader_election.cc:290] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.734040 4404 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.734566 4404 raft_consensus.cc:695] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 1 LEADER]: Becoming Leader. State: Replica: ceb8e073375545dfb45f34e0560280db, State: Running, Role: LEADER
I20250904 07:53:42.734655 4401 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:42.734786 4404 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER }
I20250904 07:53:42.735466 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb8e073375545dfb45f34e0560280db. Latest consensus state: current_term: 1 leader_uuid: "ceb8e073375545dfb45f34e0560280db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER } }
I20250904 07:53:42.735548 4405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb8e073375545dfb45f34e0560280db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb8e073375545dfb45f34e0560280db" member_type: VOTER } }
I20250904 07:53:42.735643 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.735647 4405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:42.736155 4409 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:42.738328 4409 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:42.740506 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:42.741402 4409 catalog_manager.cc:1349] Generated new cluster ID: e5dc1306c2f94f12ac09a27e48373836
I20250904 07:53:42.741468 4409 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:42.754128 4409 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:42.754923 4409 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:42.771575 4409 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db: Generated new TSK 0
I20250904 07:53:42.771781 4409 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:42.805227 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:42.807691 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.808179 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:42.808243 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:42.808498 413 server_base.cc:1047] running on GCE node
I20250904 07:53:42.808743 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:42.808796 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:42.808832 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972422808828 us; error 0 us; skew 500 ppm
I20250904 07:53:42.809005 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:42.810093 413 webserver.cc:480] Webserver started at http://127.0.103.65:39137/ using document root <none> and password file <none>
I20250904 07:53:42.810284 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:42.810352 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:42.810475 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:42.811046 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "723188224e1b4c34a17b971ad8f510e0"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.813299 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:42.814649 4430 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.814930 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:42.815033 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "723188224e1b4c34a17b971ad8f510e0"
format_stamp: "Formatted at 2025-09-04 07:53:42 on dist-test-slave-ghlz"
I20250904 07:53:42.815135 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestInsertEmptyPK.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:42.823621 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:42.824106 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:42.825629 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:42.825714 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.825798 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:42.825855 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:42.851022 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:36855
I20250904 07:53:42.851051 4500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:36855 every 8 connection(s)
I20250904 07:53:42.853277 4501 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35647
I20250904 07:53:42.853386 4501 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:42.853662 4501 heartbeater.cc:507] Master 127.0.103.126:35647 requested a full tablet report, sending...
I20250904 07:53:42.854489 4356 ts_manager.cc:194] Registered new tserver with Master: 723188224e1b4c34a17b971ad8f510e0 (127.0.103.65:36855)
I20250904 07:53:42.854857 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002301201s
I20250904 07:53:42.855485 4356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33076
I20250904 07:53:42.864408 4356 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33098:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:42.866765 4501 heartbeater.cc:499] Master 127.0.103.126:35647 was elected leader, sending a full tablet report...
I20250904 07:53:42.876495 4461 tablet_service.cc:1468] Processing CreateTablet for tablet 3b6dc18e17ca46bab57f89fef0f47f2b (DEFAULT_TABLE table=client-testtb [id=395abd117a3441108629eb05da629c82]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:42.876535 4460 tablet_service.cc:1468] Processing CreateTablet for tablet 9eaa1197c975444683e026ea935bf55c (DEFAULT_TABLE table=client-testtb [id=395abd117a3441108629eb05da629c82]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:42.876976 4461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6dc18e17ca46bab57f89fef0f47f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.877192 4460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eaa1197c975444683e026ea935bf55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.883288 4511 tablet_bootstrap.cc:492] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: Bootstrap starting.
I20250904 07:53:42.885387 4511 tablet_bootstrap.cc:654] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.886850 4511 tablet_bootstrap.cc:492] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: No bootstrap required, opened a new log
I20250904 07:53:42.886984 4511 ts_tablet_manager.cc:1397] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:53:42.887552 4511 raft_consensus.cc:357] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.887734 4511 raft_consensus.cc:383] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.887815 4511 raft_consensus.cc:738] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.888036 4511 consensus_queue.cc:260] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.888159 4511 raft_consensus.cc:397] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.888244 4511 raft_consensus.cc:491] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.888334 4511 raft_consensus.cc:3058] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.890641 4511 raft_consensus.cc:513] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.890836 4511 leader_election.cc:304] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723188224e1b4c34a17b971ad8f510e0; no voters:
I20250904 07:53:42.891220 4511 leader_election.cc:290] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.891362 4513 raft_consensus.cc:2802] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.891793 4513 raft_consensus.cc:695] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Becoming Leader. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Running, Role: LEADER
I20250904 07:53:42.891849 4511 ts_tablet_manager.cc:1428] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250904 07:53:42.892134 4511 tablet_bootstrap.cc:492] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: Bootstrap starting.
I20250904 07:53:42.892066 4513 consensus_queue.cc:237] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.893962 4511 tablet_bootstrap.cc:654] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.894080 4356 catalog_manager.cc:5582] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 723188224e1b4c34a17b971ad8f510e0 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "723188224e1b4c34a17b971ad8f510e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.897759 4511 tablet_bootstrap.cc:492] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: No bootstrap required, opened a new log
I20250904 07:53:42.897893 4511 ts_tablet_manager.cc:1397] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250904 07:53:42.898671 4511 raft_consensus.cc:357] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.898847 4511 raft_consensus.cc:383] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.898936 4511 raft_consensus.cc:738] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.899142 4511 consensus_queue.cc:260] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.899269 4511 raft_consensus.cc:397] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.899349 4511 raft_consensus.cc:491] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.899435 4511 raft_consensus.cc:3058] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.901196 4511 raft_consensus.cc:513] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.901389 4511 leader_election.cc:304] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723188224e1b4c34a17b971ad8f510e0; no voters:
I20250904 07:53:42.901589 4511 leader_election.cc:290] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.901654 4513 raft_consensus.cc:2802] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.901844 4513 raft_consensus.cc:695] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Becoming Leader. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Running, Role: LEADER
I20250904 07:53:42.902027 4511 ts_tablet_manager.cc:1428] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:42.902101 4513 consensus_queue.cc:237] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.903863 4356 catalog_manager.cc:5582] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 723188224e1b4c34a17b971ad8f510e0 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "723188224e1b4c34a17b971ad8f510e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:42.912660 4356 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33098:
name: "empty-pk"
schema {
columns {
name: "k1"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "v1"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "k1"
}
}
}
I20250904 07:53:42.918088 4461 tablet_service.cc:1468] Processing CreateTablet for tablet 46f70eb55e4645c588c95f8a19c05a9b (DEFAULT_TABLE table=empty-pk [id=f745be1f8703458dab6bfcba007bdc32]), partition=RANGE (k1) PARTITION UNBOUNDED
I20250904 07:53:42.918526 4461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f70eb55e4645c588c95f8a19c05a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:42.922454 4511 tablet_bootstrap.cc:492] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: Bootstrap starting.
I20250904 07:53:42.923723 4511 tablet_bootstrap.cc:654] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:42.925233 4511 tablet_bootstrap.cc:492] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: No bootstrap required, opened a new log
I20250904 07:53:42.925367 4511 ts_tablet_manager.cc:1397] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:42.925871 4511 raft_consensus.cc:357] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.925997 4511 raft_consensus.cc:383] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:42.926069 4511 raft_consensus.cc:738] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Initialized, Role: FOLLOWER
I20250904 07:53:42.926261 4511 consensus_queue.cc:260] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.926378 4511 raft_consensus.cc:397] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:42.926450 4511 raft_consensus.cc:491] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:42.926551 4511 raft_consensus.cc:3058] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:42.928378 4511 raft_consensus.cc:513] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.928566 4511 leader_election.cc:304] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723188224e1b4c34a17b971ad8f510e0; no voters:
I20250904 07:53:42.928726 4511 leader_election.cc:290] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:42.928788 4513 raft_consensus.cc:2802] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:42.929020 4513 raft_consensus.cc:695] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Becoming Leader. State: Replica: 723188224e1b4c34a17b971ad8f510e0, State: Running, Role: LEADER
I20250904 07:53:42.929162 4511 ts_tablet_manager.cc:1428] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:42.929306 4513 consensus_queue.cc:237] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } }
I20250904 07:53:42.931015 4355 catalog_manager.cc:5582] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 723188224e1b4c34a17b971ad8f510e0 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "723188224e1b4c34a17b971ad8f510e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723188224e1b4c34a17b971ad8f510e0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36855 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.047345 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:43.057013 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:43.057538 413 tablet_replica.cc:331] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0: stopping tablet replica
I20250904 07:53:43.057747 413 raft_consensus.cc:2241] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.057976 413 raft_consensus.cc:2270] T 46f70eb55e4645c588c95f8a19c05a9b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.058979 413 tablet_replica.cc:331] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0: stopping tablet replica
I20250904 07:53:43.059124 413 raft_consensus.cc:2241] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.059269 413 raft_consensus.cc:2270] T 3b6dc18e17ca46bab57f89fef0f47f2b P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.060050 413 tablet_replica.cc:331] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0: stopping tablet replica
I20250904 07:53:43.060202 413 raft_consensus.cc:2241] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.060348 413 raft_consensus.cc:2270] T 9eaa1197c975444683e026ea935bf55c P 723188224e1b4c34a17b971ad8f510e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.075430 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:43.081017 413 master.cc:561] Master@127.0.103.126:35647 shutting down...
I20250904 07:53:43.090644 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.090862 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.091033 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P ceb8e073375545dfb45f34e0560280db: stopping tablet replica
I20250904 07:53:43.105331 413 master.cc:583] Master@127.0.103.126:35647 shutdown complete.
[ OK ] ClientTest.TestInsertEmptyPK (438 ms)
[ RUN ] ClientTest.TestClonePredicates
I20250904 07:53:43.122392 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46327
I20250904 07:53:43.122865 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.125200 4529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.125413 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.125910 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.126470 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.126705 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.126766 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.126804 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423126801 us; error 0 us; skew 500 ppm
I20250904 07:53:43.126981 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.128187 413 webserver.cc:480] Webserver started at http://127.0.103.126:39017/ using document root <none> and password file <none>
I20250904 07:53:43.128400 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.128468 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.128579 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.129330 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "66b8914a376546fcb21ddd6c1d6388c6"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.132172 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250904 07:53:43.133906 4537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.134212 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:43.134295 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "66b8914a376546fcb21ddd6c1d6388c6"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.134387 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.147204 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.147626 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.163005 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46327
I20250904 07:53:43.163010 4598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46327 every 8 connection(s)
I20250904 07:53:43.164289 4599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.169612 4599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6: Bootstrap starting.
I20250904 07:53:43.170960 4599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.172475 4599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6: No bootstrap required, opened a new log
I20250904 07:53:43.173194 4599 raft_consensus.cc:357] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER }
I20250904 07:53:43.173342 4599 raft_consensus.cc:383] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.173384 4599 raft_consensus.cc:738] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b8914a376546fcb21ddd6c1d6388c6, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.173549 4599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER }
I20250904 07:53:43.173674 4599 raft_consensus.cc:397] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.173724 4599 raft_consensus.cc:491] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.173789 4599 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.176119 4599 raft_consensus.cc:513] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER }
I20250904 07:53:43.176306 4599 leader_election.cc:304] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b8914a376546fcb21ddd6c1d6388c6; no voters:
I20250904 07:53:43.176724 4599 leader_election.cc:290] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.176797 4602 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.177480 4602 raft_consensus.cc:695] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 1 LEADER]: Becoming Leader. State: Replica: 66b8914a376546fcb21ddd6c1d6388c6, State: Running, Role: LEADER
I20250904 07:53:43.177503 4599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:43.177747 4602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER }
I20250904 07:53:43.178458 4603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b8914a376546fcb21ddd6c1d6388c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER } }
I20250904 07:53:43.178603 4603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.178828 4604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b8914a376546fcb21ddd6c1d6388c6. Latest consensus state: current_term: 1 leader_uuid: "66b8914a376546fcb21ddd6c1d6388c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b8914a376546fcb21ddd6c1d6388c6" member_type: VOTER } }
I20250904 07:53:43.178993 4604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.189653 4608 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:43.190858 4608 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:43.193392 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:43.194214 4608 catalog_manager.cc:1349] Generated new cluster ID: b5652c71cda0465eaf213451953e8e8d
I20250904 07:53:43.194298 4608 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:43.220299 4608 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:43.221329 4608 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:43.230244 4608 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6: Generated new TSK 0
I20250904 07:53:43.230468 4608 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:43.258677 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.261404 4620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.261509 4621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.262279 4623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.262809 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.263072 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.263124 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.263146 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423263144 us; error 0 us; skew 500 ppm
I20250904 07:53:43.263288 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.264519 413 webserver.cc:480] Webserver started at http://127.0.103.65:46499/ using document root <none> and password file <none>
I20250904 07:53:43.264705 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.264762 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.264901 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.265496 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "7453fae1eb1042aa967ce26b60b6ee1a"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.267807 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:53:43.269454 4628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.269799 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:43.269898 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "7453fae1eb1042aa967ce26b60b6ee1a"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.270007 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestClonePredicates.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.284965 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.285399 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.287060 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:43.287122 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.287209 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:43.287261 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.312249 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:33007
I20250904 07:53:43.312288 4698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:33007 every 8 connection(s)
I20250904 07:53:43.314649 4699 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46327
I20250904 07:53:43.314767 4699 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:43.315109 4699 heartbeater.cc:507] Master 127.0.103.126:46327 requested a full tablet report, sending...
I20250904 07:53:43.315862 4554 ts_manager.cc:194] Registered new tserver with Master: 7453fae1eb1042aa967ce26b60b6ee1a (127.0.103.65:33007)
I20250904 07:53:43.316237 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002265663s
I20250904 07:53:43.317124 4554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48316
I20250904 07:53:43.327457 4554 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48326:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:43.328460 4699 heartbeater.cc:499] Master 127.0.103.126:46327 was elected leader, sending a full tablet report...
I20250904 07:53:43.340819 4659 tablet_service.cc:1468] Processing CreateTablet for tablet 6294f9cd56124e95a4c3828ca30012bd (DEFAULT_TABLE table=client-testtb [id=70f31ab7f0a0431aa444e7cc0a7b83f4]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:43.341003 4658 tablet_service.cc:1468] Processing CreateTablet for tablet af4f3889265748b191005ad40e38ab2f (DEFAULT_TABLE table=client-testtb [id=70f31ab7f0a0431aa444e7cc0a7b83f4]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:43.341290 4659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6294f9cd56124e95a4c3828ca30012bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.341564 4658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4f3889265748b191005ad40e38ab2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.347569 4709 tablet_bootstrap.cc:492] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: Bootstrap starting.
I20250904 07:53:43.348673 4709 tablet_bootstrap.cc:654] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.350025 4709 tablet_bootstrap.cc:492] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: No bootstrap required, opened a new log
I20250904 07:53:43.350116 4709 ts_tablet_manager.cc:1397] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:43.350577 4709 raft_consensus.cc:357] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.350688 4709 raft_consensus.cc:383] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.350726 4709 raft_consensus.cc:738] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7453fae1eb1042aa967ce26b60b6ee1a, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.350885 4709 consensus_queue.cc:260] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.350996 4709 raft_consensus.cc:397] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.351035 4709 raft_consensus.cc:491] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.351082 4709 raft_consensus.cc:3058] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.353036 4709 raft_consensus.cc:513] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.353189 4709 leader_election.cc:304] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7453fae1eb1042aa967ce26b60b6ee1a; no voters:
I20250904 07:53:43.353492 4709 leader_election.cc:290] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.353624 4711 raft_consensus.cc:2802] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.354003 4709 ts_tablet_manager.cc:1428] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250904 07:53:43.354200 4711 raft_consensus.cc:695] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 LEADER]: Becoming Leader. State: Replica: 7453fae1eb1042aa967ce26b60b6ee1a, State: Running, Role: LEADER
I20250904 07:53:43.354218 4709 tablet_bootstrap.cc:492] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: Bootstrap starting.
I20250904 07:53:43.354491 4711 consensus_queue.cc:237] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.355412 4709 tablet_bootstrap.cc:654] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.356719 4553 catalog_manager.cc:5582] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7453fae1eb1042aa967ce26b60b6ee1a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.358911 4709 tablet_bootstrap.cc:492] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: No bootstrap required, opened a new log
I20250904 07:53:43.359058 4709 ts_tablet_manager.cc:1397] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250904 07:53:43.359643 4709 raft_consensus.cc:357] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.359747 4709 raft_consensus.cc:383] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.359777 4709 raft_consensus.cc:738] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7453fae1eb1042aa967ce26b60b6ee1a, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.359961 4709 consensus_queue.cc:260] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.360085 4709 raft_consensus.cc:397] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.360144 4709 raft_consensus.cc:491] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.360220 4709 raft_consensus.cc:3058] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.362049 4709 raft_consensus.cc:513] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.362195 4709 leader_election.cc:304] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7453fae1eb1042aa967ce26b60b6ee1a; no voters:
I20250904 07:53:43.362303 4709 leader_election.cc:290] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.362367 4711 raft_consensus.cc:2802] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.362502 4711 raft_consensus.cc:695] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 LEADER]: Becoming Leader. State: Replica: 7453fae1eb1042aa967ce26b60b6ee1a, State: Running, Role: LEADER
I20250904 07:53:43.362588 4709 ts_tablet_manager.cc:1428] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:43.362706 4711 consensus_queue.cc:237] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } }
I20250904 07:53:43.364163 4553 catalog_manager.cc:5582] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7453fae1eb1042aa967ce26b60b6ee1a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7453fae1eb1042aa967ce26b60b6ee1a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33007 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.392652 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:43.402099 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:43.402514 413 tablet_replica.cc:331] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a: stopping tablet replica
I20250904 07:53:43.402666 413 raft_consensus.cc:2241] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.402832 413 raft_consensus.cc:2270] T af4f3889265748b191005ad40e38ab2f P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.403549 413 tablet_replica.cc:331] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a: stopping tablet replica
I20250904 07:53:43.403671 413 raft_consensus.cc:2241] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.403818 413 raft_consensus.cc:2270] T 6294f9cd56124e95a4c3828ca30012bd P 7453fae1eb1042aa967ce26b60b6ee1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.408093 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:43.412052 413 master.cc:561] Master@127.0.103.126:46327 shutting down...
I20250904 07:53:43.421052 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.421253 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.421507 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 66b8914a376546fcb21ddd6c1d6388c6: stopping tablet replica
I20250904 07:53:43.425334 413 master.cc:583] Master@127.0.103.126:46327 shutdown complete.
[ OK ] ClientTest.TestClonePredicates (320 ms)
[ RUN ] ClientTest.TestErrorCollector
I20250904 07:53:43.442687 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46589
I20250904 07:53:43.443086 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.445091 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.445158 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.445516 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.446504 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.446727 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.446777 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.446818 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423446813 us; error 0 us; skew 500 ppm
I20250904 07:53:43.446971 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.447957 413 webserver.cc:480] Webserver started at http://127.0.103.126:38463/ using document root <none> and password file <none>
I20250904 07:53:43.448143 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.448202 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.448313 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.448846 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.450989 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:43.452303 4728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.452567 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:43.452657 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.452746 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.458812 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.459163 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.469878 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46589
I20250904 07:53:43.469898 4789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46589 every 8 connection(s)
I20250904 07:53:43.470903 4790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.474684 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b: Bootstrap starting.
I20250904 07:53:43.475679 4790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.476925 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b: No bootstrap required, opened a new log
I20250904 07:53:43.477409 4790 raft_consensus.cc:357] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER }
I20250904 07:53:43.477514 4790 raft_consensus.cc:383] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.477548 4790 raft_consensus.cc:738] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ed2c542a3f49a6b6fb3003d19ccc0b, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.477702 4790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER }
I20250904 07:53:43.477809 4790 raft_consensus.cc:397] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.477849 4790 raft_consensus.cc:491] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.477900 4790 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.479496 4790 raft_consensus.cc:513] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER }
I20250904 07:53:43.479643 4790 leader_election.cc:304] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ed2c542a3f49a6b6fb3003d19ccc0b; no voters:
I20250904 07:53:43.479928 4790 leader_election.cc:290] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.480041 4793 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.480448 4793 raft_consensus.cc:695] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 1 LEADER]: Becoming Leader. State: Replica: 04ed2c542a3f49a6b6fb3003d19ccc0b, State: Running, Role: LEADER
I20250904 07:53:43.480645 4790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:43.480639 4793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER }
I20250904 07:53:43.481400 4795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ed2c542a3f49a6b6fb3003d19ccc0b. Latest consensus state: current_term: 1 leader_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER } }
I20250904 07:53:43.481539 4795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.481447 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ed2c542a3f49a6b6fb3003d19ccc0b" member_type: VOTER } }
I20250904 07:53:43.481781 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.482973 4798 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:43.484766 4798 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:43.485337 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:43.487360 4798 catalog_manager.cc:1349] Generated new cluster ID: e55c3b62d8924b3c8b953a5137c97503
I20250904 07:53:43.487432 4798 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:43.495677 4798 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:43.496335 4798 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:43.504843 4798 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b: Generated new TSK 0
I20250904 07:53:43.505048 4798 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:43.517731 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.520000 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.520318 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.520397 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.520539 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.520884 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.520948 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.520972 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423520969 us; error 0 us; skew 500 ppm
I20250904 07:53:43.521065 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.522028 413 webserver.cc:480] Webserver started at http://127.0.103.65:36487/ using document root <none> and password file <none>
I20250904 07:53:43.522194 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.522240 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.522315 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.522783 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "30e73176e91941a690ccbfc89ab929e9"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.524812 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:43.526171 4819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.526438 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:43.526526 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "30e73176e91941a690ccbfc89ab929e9"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.526611 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestErrorCollector.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.547406 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.547817 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.549213 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:43.549288 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.549358 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:43.549398 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.565821 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38277
I20250904 07:53:43.565848 4889 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38277 every 8 connection(s)
I20250904 07:53:43.567124 4890 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46589
I20250904 07:53:43.567229 4890 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:43.567499 4890 heartbeater.cc:507] Master 127.0.103.126:46589 requested a full tablet report, sending...
I20250904 07:53:43.568188 4745 ts_manager.cc:194] Registered new tserver with Master: 30e73176e91941a690ccbfc89ab929e9 (127.0.103.65:38277)
I20250904 07:53:43.568845 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.0022705s
I20250904 07:53:43.569114 4745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37384
I20250904 07:53:43.577716 4745 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37404:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:43.580159 4890 heartbeater.cc:499] Master 127.0.103.126:46589 was elected leader, sending a full tablet report...
I20250904 07:53:43.593891 4851 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6310f57db74cf7acd621beb0fb75bd (DEFAULT_TABLE table=client-testtb [id=54bf8da0696c484b8937ddf05dd76b3b]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:43.593925 4849 tablet_service.cc:1468] Processing CreateTablet for tablet 43a909f219a24a089be9ce0c203b6a7b (DEFAULT_TABLE table=client-testtb [id=54bf8da0696c484b8937ddf05dd76b3b]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:43.594283 4851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6310f57db74cf7acd621beb0fb75bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.594511 4849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43a909f219a24a089be9ce0c203b6a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.598996 4900 tablet_bootstrap.cc:492] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: Bootstrap starting.
I20250904 07:53:43.600042 4900 tablet_bootstrap.cc:654] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.601387 4900 tablet_bootstrap.cc:492] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: No bootstrap required, opened a new log
I20250904 07:53:43.601502 4900 ts_tablet_manager.cc:1397] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:43.602000 4900 raft_consensus.cc:357] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.602123 4900 raft_consensus.cc:383] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.602197 4900 raft_consensus.cc:738] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e73176e91941a690ccbfc89ab929e9, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.602392 4900 consensus_queue.cc:260] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.602638 4900 raft_consensus.cc:397] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.602741 4900 raft_consensus.cc:491] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.602818 4900 raft_consensus.cc:3058] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.604655 4900 raft_consensus.cc:513] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.604817 4900 leader_election.cc:304] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e73176e91941a690ccbfc89ab929e9; no voters:
I20250904 07:53:43.605230 4900 leader_election.cc:290] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.605356 4902 raft_consensus.cc:2802] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.605762 4902 raft_consensus.cc:695] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 1 LEADER]: Becoming Leader. State: Replica: 30e73176e91941a690ccbfc89ab929e9, State: Running, Role: LEADER
I20250904 07:53:43.605814 4900 ts_tablet_manager.cc:1428] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250904 07:53:43.605980 4902 consensus_queue.cc:237] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.606055 4900 tablet_bootstrap.cc:492] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: Bootstrap starting.
I20250904 07:53:43.607246 4900 tablet_bootstrap.cc:654] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.607784 4745 catalog_manager.cc:5582] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e73176e91941a690ccbfc89ab929e9 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "30e73176e91941a690ccbfc89ab929e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.609900 4900 tablet_bootstrap.cc:492] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: No bootstrap required, opened a new log
I20250904 07:53:43.610054 4900 ts_tablet_manager.cc:1397] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:43.610893 4900 raft_consensus.cc:357] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.611074 4900 raft_consensus.cc:383] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.611140 4900 raft_consensus.cc:738] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e73176e91941a690ccbfc89ab929e9, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.611364 4900 consensus_queue.cc:260] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.611527 4900 raft_consensus.cc:397] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.611601 4900 raft_consensus.cc:491] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.611696 4900 raft_consensus.cc:3058] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.614316 4900 raft_consensus.cc:513] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.614473 4900 leader_election.cc:304] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e73176e91941a690ccbfc89ab929e9; no voters:
I20250904 07:53:43.614637 4900 leader_election.cc:290] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.614701 4902 raft_consensus.cc:2802] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.614869 4902 raft_consensus.cc:695] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 1 LEADER]: Becoming Leader. State: Replica: 30e73176e91941a690ccbfc89ab929e9, State: Running, Role: LEADER
I20250904 07:53:43.615007 4900 ts_tablet_manager.cc:1428] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250904 07:53:43.615069 4902 consensus_queue.cc:237] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } }
I20250904 07:53:43.616688 4745 catalog_manager.cc:5582] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30e73176e91941a690ccbfc89ab929e9 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "30e73176e91941a690ccbfc89ab929e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e73176e91941a690ccbfc89ab929e9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38277 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.640878 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:43.649282 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:43.649570 413 tablet_replica.cc:331] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9: stopping tablet replica
I20250904 07:53:43.649751 413 raft_consensus.cc:2241] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.649904 413 raft_consensus.cc:2270] T 1a6310f57db74cf7acd621beb0fb75bd P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.650615 413 tablet_replica.cc:331] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9: stopping tablet replica
I20250904 07:53:43.650712 413 raft_consensus.cc:2241] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.650813 413 raft_consensus.cc:2270] T 43a909f219a24a089be9ce0c203b6a7b P 30e73176e91941a690ccbfc89ab929e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.665307 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:43.669363 413 master.cc:561] Master@127.0.103.126:46589 shutting down...
I20250904 07:53:43.677590 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.677758 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.677917 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 04ed2c542a3f49a6b6fb3003d19ccc0b: stopping tablet replica
I20250904 07:53:43.681962 413 master.cc:583] Master@127.0.103.126:46589 shutdown complete.
[ OK ] ClientTest.TestErrorCollector (248 ms)
[ RUN ] ClientTest.TestBlockScannerHijackingAttempts
I20250904 07:53:43.691236 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39353
I20250904 07:53:43.691661 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.693660 4912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.693656 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.694759 4914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.698006 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.698297 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.698359 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.698396 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423698392 us; error 0 us; skew 500 ppm
I20250904 07:53:43.698575 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.699885 413 webserver.cc:480] Webserver started at http://127.0.103.126:36799/ using document root <none> and password file <none>
I20250904 07:53:43.700114 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.700191 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.700312 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.701117 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "d224d3079a304241af2c3efe0f993a61"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.704120 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250904 07:53:43.705689 4919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.706014 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:43.706101 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d224d3079a304241af2c3efe0f993a61"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.706207 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.726056 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.726495 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.737463 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39353
I20250904 07:53:43.737484 4980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39353 every 8 connection(s)
I20250904 07:53:43.738756 4981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.742589 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Bootstrap starting.
I20250904 07:53:43.743626 4981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.744953 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: No bootstrap required, opened a new log
I20250904 07:53:43.745467 4981 raft_consensus.cc:357] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.745577 4981 raft_consensus.cc:383] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.745632 4981 raft_consensus.cc:738] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d224d3079a304241af2c3efe0f993a61, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.745816 4981 consensus_queue.cc:260] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.745914 4981 raft_consensus.cc:397] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.745985 4981 raft_consensus.cc:491] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.746065 4981 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.747710 4981 raft_consensus.cc:513] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.747875 4981 leader_election.cc:304] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d224d3079a304241af2c3efe0f993a61; no voters:
I20250904 07:53:43.748229 4981 leader_election.cc:290] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.748313 4984 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.748718 4984 raft_consensus.cc:695] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 LEADER]: Becoming Leader. State: Replica: d224d3079a304241af2c3efe0f993a61, State: Running, Role: LEADER
I20250904 07:53:43.748940 4984 consensus_queue.cc:237] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.748999 4981 sys_catalog.cc:564] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:43.749665 4986 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d224d3079a304241af2c3efe0f993a61. Latest consensus state: current_term: 1 leader_uuid: "d224d3079a304241af2c3efe0f993a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER } }
I20250904 07:53:43.749802 4986 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.749727 4985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d224d3079a304241af2c3efe0f993a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER } }
I20250904 07:53:43.749881 4985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.750977 4989 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:43.752120 4989 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:43.754737 4989 catalog_manager.cc:1349] Generated new cluster ID: 2dfb2db7902944929d5ca599543ed204
I20250904 07:53:43.754810 4989 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:43.756321 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:43.767268 4989 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:43.767971 4989 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:43.775350 4989 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Generated new TSK 0
I20250904 07:53:43.775540 4989 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:43.788775 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.791194 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.791251 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.791608 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.792474 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.792712 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.792762 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.792784 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423792782 us; error 0 us; skew 500 ppm
I20250904 07:53:43.792922 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.793980 413 webserver.cc:480] Webserver started at http://127.0.103.65:45501/ using document root <none> and password file <none>
I20250904 07:53:43.794140 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.794196 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:43.794279 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:43.794795 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "3a5bb0f2074f4d6192365aaf2d055217"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.796926 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:43.798326 5010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.798648 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:43.798729 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "3a5bb0f2074f4d6192365aaf2d055217"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.798820 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.811357 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.811753 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.813163 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:43.813248 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.813345 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:43.813412 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.833993 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39393
I20250904 07:53:43.834019 5080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39393 every 8 connection(s)
I20250904 07:53:43.835343 5081 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39353
I20250904 07:53:43.835451 5081 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:43.835740 5081 heartbeater.cc:507] Master 127.0.103.126:39353 requested a full tablet report, sending...
I20250904 07:53:43.836567 4936 ts_manager.cc:194] Registered new tserver with Master: 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65:39393)
I20250904 07:53:43.837087 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002276145s
I20250904 07:53:43.837628 4936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37094
I20250904 07:53:43.846657 4936 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37108:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:43.848667 5081 heartbeater.cc:499] Master 127.0.103.126:39353 was elected leader, sending a full tablet report...
I20250904 07:53:43.857954 5041 tablet_service.cc:1468] Processing CreateTablet for tablet df2ea2930fb54f97b14af2b215afe7cc (DEFAULT_TABLE table=client-testtb [id=46bdc8c0b4f04874b7e5bacde838c1a1]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:43.858021 5040 tablet_service.cc:1468] Processing CreateTablet for tablet 4a762537560f4278bd5a8011cf7c3fbe (DEFAULT_TABLE table=client-testtb [id=46bdc8c0b4f04874b7e5bacde838c1a1]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:43.858342 5041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df2ea2930fb54f97b14af2b215afe7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.858510 5040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a762537560f4278bd5a8011cf7c3fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:43.864503 5091 tablet_bootstrap.cc:492] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap starting.
I20250904 07:53:43.865685 5091 tablet_bootstrap.cc:654] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.867043 5091 tablet_bootstrap.cc:492] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: No bootstrap required, opened a new log
I20250904 07:53:43.867136 5091 ts_tablet_manager.cc:1397] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:43.867580 5091 raft_consensus.cc:357] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.867697 5091 raft_consensus.cc:383] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.867735 5091 raft_consensus.cc:738] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.867900 5091 consensus_queue.cc:260] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.868031 5091 raft_consensus.cc:397] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.868072 5091 raft_consensus.cc:491] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.868124 5091 raft_consensus.cc:3058] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.869753 5091 raft_consensus.cc:513] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.869922 5091 leader_election.cc:304] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5bb0f2074f4d6192365aaf2d055217; no voters:
I20250904 07:53:43.870316 5091 leader_election.cc:290] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.870432 5093 raft_consensus.cc:2802] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.871017 5091 ts_tablet_manager.cc:1428] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:43.871261 5091 tablet_bootstrap.cc:492] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap starting.
I20250904 07:53:43.872255 5091 tablet_bootstrap.cc:654] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:43.873551 5091 tablet_bootstrap.cc:492] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: No bootstrap required, opened a new log
I20250904 07:53:43.873664 5091 ts_tablet_manager.cc:1397] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:43.874106 5091 raft_consensus.cc:357] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.874225 5091 raft_consensus.cc:383] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:43.874287 5091 raft_consensus.cc:738] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.874468 5091 consensus_queue.cc:260] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.874578 5091 raft_consensus.cc:397] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.874639 5091 raft_consensus.cc:491] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.876976 5093 raft_consensus.cc:695] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Running, Role: LEADER
I20250904 07:53:43.877163 5093 consensus_queue.cc:237] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.878962 4936 catalog_manager.cc:5582] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "3a5bb0f2074f4d6192365aaf2d055217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.883673 5091 raft_consensus.cc:3058] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:43.885401 5091 raft_consensus.cc:513] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.885574 5091 leader_election.cc:304] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5bb0f2074f4d6192365aaf2d055217; no voters:
I20250904 07:53:43.885713 5091 leader_election.cc:290] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:43.885774 5093 raft_consensus.cc:2802] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:43.885944 5093 raft_consensus.cc:695] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Running, Role: LEADER
I20250904 07:53:43.886065 5091 ts_tablet_manager.cc:1428] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20250904 07:53:43.886128 5093 consensus_queue.cc:237] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:43.887677 4936 catalog_manager.cc:5582] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "3a5bb0f2074f4d6192365aaf2d055217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:43.893590 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:43.900471 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:43.900722 413 tablet_replica.cc:331] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: stopping tablet replica
I20250904 07:53:43.901032 413 raft_consensus.cc:2241] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.901194 413 raft_consensus.cc:2270] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.901845 413 tablet_replica.cc:331] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: stopping tablet replica
I20250904 07:53:43.901947 413 raft_consensus.cc:2241] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.902040 413 raft_consensus.cc:2270] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.915745 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:43.919780 413 master.cc:561] Master@127.0.103.126:39353 shutting down...
I20250904 07:53:43.927735 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:43.927898 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:43.928035 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: stopping tablet replica
I20250904 07:53:43.931887 413 master.cc:583] Master@127.0.103.126:39353 shutdown complete.
I20250904 07:53:43.935761 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.937705 5100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.937808 5101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.938139 5103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.938290 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.938551 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.938614 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.938652 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423938649 us; error 0 us; skew 500 ppm
I20250904 07:53:43.938807 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.939806 413 webserver.cc:480] Webserver started at http://127.0.103.126:33215/ using document root <none> and password file <none>
I20250904 07:53:43.939975 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.940016 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:43.941704 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:43.942773 5108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:43.943063 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:43.943173 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d224d3079a304241af2c3efe0f993a61"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:43.943269 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:43.953338 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:43.953747 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:43.964406 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39353
I20250904 07:53:43.964432 5169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39353 every 8 connection(s)
I20250904 07:53:43.967691 5170 sys_catalog.cc:263] Verifying existing consensus state
I20250904 07:53:43.969477 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Bootstrap starting.
I20250904 07:53:43.978868 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:43.979302 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Bootstrap complete.
I20250904 07:53:43.979941 5170 raft_consensus.cc:357] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.980096 5170 raft_consensus.cc:738] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d224d3079a304241af2c3efe0f993a61, State: Initialized, Role: FOLLOWER
I20250904 07:53:43.980268 5170 consensus_queue.cc:260] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.980378 5170 raft_consensus.cc:397] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:43.980436 5170 raft_consensus.cc:491] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:43.980518 5170 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:43.982277 5170 raft_consensus.cc:513] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.982438 5170 leader_election.cc:304] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d224d3079a304241af2c3efe0f993a61; no voters:
I20250904 07:53:43.982787 5170 leader_election.cc:290] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 07:53:43.982899 5173 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:43.983304 5173 raft_consensus.cc:695] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 2 LEADER]: Becoming Leader. State: Replica: d224d3079a304241af2c3efe0f993a61, State: Running, Role: LEADER
I20250904 07:53:43.983493 5173 consensus_queue.cc:237] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER }
I20250904 07:53:43.983569 5170 sys_catalog.cc:564] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:43.984119 5174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d224d3079a304241af2c3efe0f993a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER } }
I20250904 07:53:43.984187 5175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d224d3079a304241af2c3efe0f993a61. Latest consensus state: current_term: 2 leader_uuid: "d224d3079a304241af2c3efe0f993a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224d3079a304241af2c3efe0f993a61" member_type: VOTER } }
I20250904 07:53:43.984346 5175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.984323 5174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:43.985328 5179 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:43.986558 5179 catalog_manager.cc:671] Loaded metadata for table client-testtb [id=46bdc8c0b4f04874b7e5bacde838c1a1]
I20250904 07:53:43.987671 5179 tablet_loader.cc:96] loaded metadata for tablet 4a762537560f4278bd5a8011cf7c3fbe (table client-testtb [id=46bdc8c0b4f04874b7e5bacde838c1a1])
I20250904 07:53:43.988131 5179 tablet_loader.cc:96] loaded metadata for tablet df2ea2930fb54f97b14af2b215afe7cc (table client-testtb [id=46bdc8c0b4f04874b7e5bacde838c1a1])
I20250904 07:53:43.988373 5179 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:43.989181 5179 catalog_manager.cc:1261] Loaded cluster ID: 2dfb2db7902944929d5ca599543ed204
I20250904 07:53:43.989245 5179 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:43.990849 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:43.991366 5179 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:43.992049 5179 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61: Loaded TSK: 0
I20250904 07:53:43.992591 5179 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:43.994704 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:43.997009 5191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.997051 5190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:43.997557 5193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:43.998057 413 server_base.cc:1047] running on GCE node
I20250904 07:53:43.998263 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:43.998317 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:43.998365 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972423998361 us; error 0 us; skew 500 ppm
I20250904 07:53:43.998507 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:43.999557 413 webserver.cc:480] Webserver started at http://127.0.103.65:43661/ using document root <none> and password file <none>
I20250904 07:53:43.999755 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:43.999809 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:44.001449 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:44.002581 5198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:44.002871 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:44.002985 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "3a5bb0f2074f4d6192365aaf2d055217"
format_stamp: "Formatted at 2025-09-04 07:53:43 on dist-test-slave-ghlz"
I20250904 07:53:44.003087 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TestBlockScannerHijackingAttempts.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:44.028209 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:44.028631 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:44.031224 5208 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:53:44.035890 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:44.036010 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20250904 07:53:44.036106 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:53:44.038921 5208 tablet_bootstrap.cc:492] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap starting.
I20250904 07:53:44.040647 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:44.040711 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.000s
I20250904 07:53:44.045070 5208 tablet_bootstrap.cc:492] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:44.045516 5208 tablet_bootstrap.cc:492] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap complete.
I20250904 07:53:44.045727 5208 ts_tablet_manager.cc:1397] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250904 07:53:44.046272 5208 raft_consensus.cc:357] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.046424 5208 raft_consensus.cc:738] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.046644 5208 consensus_queue.cc:260] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.046815 5208 raft_consensus.cc:397] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.046866 5208 raft_consensus.cc:491] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.046914 5208 raft_consensus.cc:3058] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:44.048669 5208 raft_consensus.cc:513] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.048841 5208 leader_election.cc:304] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5bb0f2074f4d6192365aaf2d055217; no voters:
I20250904 07:53:44.049331 5208 leader_election.cc:290] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 07:53:44.049419 5220 raft_consensus.cc:2802] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:44.050069 5208 ts_tablet_manager.cc:1428] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250904 07:53:44.050376 5208 tablet_bootstrap.cc:492] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap starting.
I20250904 07:53:44.053053 5220 raft_consensus.cc:695] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 LEADER]: Becoming Leader. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Running, Role: LEADER
I20250904 07:53:44.053277 5220 consensus_queue.cc:237] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.062637 5208 tablet_bootstrap.cc:492] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:44.063150 5208 tablet_bootstrap.cc:492] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Bootstrap complete.
I20250904 07:53:44.063419 5208 ts_tablet_manager.cc:1397] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20250904 07:53:44.064257 5208 raft_consensus.cc:357] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.064441 5208 raft_consensus.cc:738] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.064749 5208 consensus_queue.cc:260] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.064952 5208 raft_consensus.cc:397] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.065032 5208 raft_consensus.cc:491] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.065122 5208 raft_consensus.cc:3058] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:44.066555 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39087
I20250904 07:53:44.066563 5276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39087 every 8 connection(s)
I20250904 07:53:44.068511 5208 raft_consensus.cc:513] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.068769 5208 leader_election.cc:304] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5bb0f2074f4d6192365aaf2d055217; no voters:
I20250904 07:53:44.069048 5208 leader_election.cc:290] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 07:53:44.069160 5220 raft_consensus.cc:2802] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:44.069326 5220 raft_consensus.cc:695] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 LEADER]: Becoming Leader. State: Replica: 3a5bb0f2074f4d6192365aaf2d055217, State: Running, Role: LEADER
I20250904 07:53:44.069427 5208 ts_tablet_manager.cc:1428] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250904 07:53:44.069551 5220 consensus_queue.cc:237] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } }
I20250904 07:53:44.070335 5277 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39353
I20250904 07:53:44.070442 5277 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:44.070773 5277 heartbeater.cc:507] Master 127.0.103.126:39353 requested a full tablet report, sending...
I20250904 07:53:44.071772 5125 ts_manager.cc:194] Registered new tserver with Master: 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65:39087)
I20250904 07:53:44.072013 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004015036s
I20250904 07:53:44.072461 5125 catalog_manager.cc:5582] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "3a5bb0f2074f4d6192365aaf2d055217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:44.072809 5125 catalog_manager.cc:5582] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "3a5bb0f2074f4d6192365aaf2d055217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5bb0f2074f4d6192365aaf2d055217" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39393 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:44.078028 5125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37118
I20250904 07:53:44.079320 5277 heartbeater.cc:499] Master 127.0.103.126:39353 was elected leader, sending a full tablet report...
I20250904 07:53:44.203341 413 client-test.cc:7842] Attempting to extract data from fault-tolerant scan eae5ce61315545f4aadc993cfaa0b585 as bad-guy
I20250904 07:53:44.203907 5231 tablet_service.cc:3073] Scan: Not authorized: User bad-guy doesn't own scanner eae5ce61315545f4aadc993cfaa0b585: call sequence id=1, remote={username='bad-guy'} at 127.0.0.1:41110
W20250904 07:53:44.204340 413 client.cc:2233] Scan on tablet server 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65:39087) with Scanner { table: client-testtb, tablet: df2ea2930fb54f97b14af2b215afe7cc, projection: (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
), scan_spec: } failed: Remote error: Not authorized: User bad-guy doesn't own scanner eae5ce61315545f4aadc993cfaa0b585
I20250904 07:53:44.205106 5231 tablet_service.cc:3073] Scan: Not authorized: User bad-guy doesn't own scanner eae5ce61315545f4aadc993cfaa0b585: call sequence id=2, remote={username='bad-guy'} at 127.0.0.1:41110
W20250904 07:53:44.205451 5291 client.cc:2040] Couldn't close scanner eae5ce61315545f4aadc993cfaa0b585: Remote error: Not authorized: User bad-guy doesn't own scanner eae5ce61315545f4aadc993cfaa0b585
I20250904 07:53:44.208797 413 client-test.cc:7842] Attempting to extract data from non-fault-tolerant scan 7e3ed16d8f3b430fb259e9213a24482e as bad-guy
I20250904 07:53:44.209393 5230 tablet_service.cc:3073] Scan: Not authorized: User bad-guy doesn't own scanner 7e3ed16d8f3b430fb259e9213a24482e: call sequence id=1, remote={username='bad-guy'} at 127.0.0.1:41110
W20250904 07:53:44.209745 413 client.cc:2233] Scan on tablet server 3a5bb0f2074f4d6192365aaf2d055217 (127.0.103.65:39087) with Scanner { table: client-testtb, tablet: df2ea2930fb54f97b14af2b215afe7cc, projection: (
10:key INT32 NOT NULL,
11:int_val INT32 NOT NULL,
12:string_val STRING NULLABLE,
13:non_null_with_default INT32 NOT NULL,
PRIMARY KEY (key)
), scan_spec: } failed: Remote error: Not authorized: User bad-guy doesn't own scanner 7e3ed16d8f3b430fb259e9213a24482e
I20250904 07:53:44.210482 5230 tablet_service.cc:3073] Scan: Not authorized: User bad-guy doesn't own scanner 7e3ed16d8f3b430fb259e9213a24482e: call sequence id=2, remote={username='bad-guy'} at 127.0.0.1:41110
W20250904 07:53:44.210888 5291 client.cc:2040] Couldn't close scanner 7e3ed16d8f3b430fb259e9213a24482e: Remote error: Not authorized: User bad-guy doesn't own scanner 7e3ed16d8f3b430fb259e9213a24482e
I20250904 07:53:44.236498 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:44.270908 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:44.271234 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:44.271415 413 raft_consensus.cc:2241] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.271615 413 raft_consensus.cc:2270] T 4a762537560f4278bd5a8011cf7c3fbe P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:44.274945 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:44.275092 413 raft_consensus.cc:2241] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.275280 413 raft_consensus.cc:2270] T df2ea2930fb54f97b14af2b215afe7cc P 3a5bb0f2074f4d6192365aaf2d055217 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:44.279655 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:44.283887 413 master.cc:561] Master@127.0.103.126:39353 shutting down...
I20250904 07:53:44.292758 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 2 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.292986 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d224d3079a304241af2c3efe0f993a61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:44.293134 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:44.306442 413 master.cc:583] Master@127.0.103.126:39353 shutdown complete.
[ OK ] ClientTest.TestBlockScannerHijackingAttempts (624 ms)
[ RUN ] ClientTest.FlushTxnSessionsOnCommit
I20250904 07:53:44.315570 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:45937
I20250904 07:53:44.316000 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:44.318075 5298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:44.318104 5299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:44.318601 413 server_base.cc:1047] running on GCE node
W20250904 07:53:44.318591 5301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:44.318982 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:44.319037 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:44.319059 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972424319057 us; error 0 us; skew 500 ppm
I20250904 07:53:44.319180 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:44.320169 413 webserver.cc:480] Webserver started at http://127.0.103.126:35703/ using document root <none> and password file <none>
I20250904 07:53:44.320348 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:44.320397 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:44.320477 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:44.321033 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "56218c64c21149118af22213d38cf65e"
format_stamp: "Formatted at 2025-09-04 07:53:44 on dist-test-slave-ghlz"
I20250904 07:53:44.323114 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:44.324321 5306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:44.324615 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:44.324697 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "56218c64c21149118af22213d38cf65e"
format_stamp: "Formatted at 2025-09-04 07:53:44 on dist-test-slave-ghlz"
I20250904 07:53:44.324791 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:44.332461 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:44.333009 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:44.343113 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:45937
I20250904 07:53:44.343133 5367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:45937 every 8 connection(s)
I20250904 07:53:44.344152 5368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:44.347827 5368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e: Bootstrap starting.
I20250904 07:53:44.348809 5368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:44.350033 5368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e: No bootstrap required, opened a new log
I20250904 07:53:44.350505 5368 raft_consensus.cc:357] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER }
I20250904 07:53:44.350608 5368 raft_consensus.cc:383] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:44.350647 5368 raft_consensus.cc:738] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56218c64c21149118af22213d38cf65e, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.350775 5368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER }
I20250904 07:53:44.350854 5368 raft_consensus.cc:397] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.350891 5368 raft_consensus.cc:491] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.350947 5368 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:44.352458 5368 raft_consensus.cc:513] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER }
I20250904 07:53:44.352595 5368 leader_election.cc:304] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56218c64c21149118af22213d38cf65e; no voters:
I20250904 07:53:44.352876 5368 leader_election.cc:290] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:44.352979 5371 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:44.353387 5371 raft_consensus.cc:695] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 1 LEADER]: Becoming Leader. State: Replica: 56218c64c21149118af22213d38cf65e, State: Running, Role: LEADER
I20250904 07:53:44.353585 5371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER }
I20250904 07:53:44.353637 5368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:44.354318 5373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56218c64c21149118af22213d38cf65e. Latest consensus state: current_term: 1 leader_uuid: "56218c64c21149118af22213d38cf65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER } }
I20250904 07:53:44.354463 5373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:44.354362 5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56218c64c21149118af22213d38cf65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56218c64c21149118af22213d38cf65e" member_type: VOTER } }
I20250904 07:53:44.354681 5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:44.355830 5376 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:44.357441 5376 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:44.358167 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:44.359959 5376 catalog_manager.cc:1349] Generated new cluster ID: 9bcbcb4cedf84370a0ef5b5e6847f814
I20250904 07:53:44.360030 5376 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:44.371206 5376 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:44.371829 5376 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:44.384028 5376 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e: Generated new TSK 0
I20250904 07:53:44.384222 5376 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:44.390832 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:44.393069 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:44.393505 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:44.393631 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:44.393864 413 server_base.cc:1047] running on GCE node
I20250904 07:53:44.394122 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:44.394179 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:44.394215 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972424394212 us; error 0 us; skew 500 ppm
I20250904 07:53:44.394346 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:44.395330 413 webserver.cc:480] Webserver started at http://127.0.103.65:40871/ using document root <none> and password file <none>
I20250904 07:53:44.395503 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:44.395558 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:44.395668 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:44.396178 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "7e76acc3b69541108fadf91060ca82b2"
format_stamp: "Formatted at 2025-09-04 07:53:44 on dist-test-slave-ghlz"
I20250904 07:53:44.398258 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:44.399461 5397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:44.399729 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:44.399835 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "7e76acc3b69541108fadf91060ca82b2"
format_stamp: "Formatted at 2025-09-04 07:53:44 on dist-test-slave-ghlz"
I20250904 07:53:44.399927 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.FlushTxnSessionsOnCommit.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:44.408902 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:44.409346 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:44.410686 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:44.410763 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:44.410845 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:44.410897 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:44.425803 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40227
I20250904 07:53:44.425828 5467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40227 every 8 connection(s)
I20250904 07:53:44.427198 5468 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45937
I20250904 07:53:44.427295 5468 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:44.427552 5468 heartbeater.cc:507] Master 127.0.103.126:45937 requested a full tablet report, sending...
I20250904 07:53:44.428258 5323 ts_manager.cc:194] Registered new tserver with Master: 7e76acc3b69541108fadf91060ca82b2 (127.0.103.65:40227)
I20250904 07:53:44.428929 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002259062s
I20250904 07:53:44.429261 5323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57242
I20250904 07:53:44.437673 5323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57270:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:44.440516 5468 heartbeater.cc:499] Master 127.0.103.126:45937 was elected leader, sending a full tablet report...
I20250904 07:53:44.448652 5431 tablet_service.cc:1468] Processing CreateTablet for tablet 0ceb771bee8541c2904d3b171a6bf6b8 (DEFAULT_TABLE table=client-testtb [id=121554c0bfa4457b80ee8af92399040b]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:44.448722 5429 tablet_service.cc:1468] Processing CreateTablet for tablet adf8a17f8cd649f9b11dfd8386e5f471 (DEFAULT_TABLE table=client-testtb [id=121554c0bfa4457b80ee8af92399040b]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:44.449057 5431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ceb771bee8541c2904d3b171a6bf6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:44.449324 5429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf8a17f8cd649f9b11dfd8386e5f471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:44.453961 5478 tablet_bootstrap.cc:492] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: Bootstrap starting.
I20250904 07:53:44.454993 5478 tablet_bootstrap.cc:654] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:44.456202 5478 tablet_bootstrap.cc:492] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: No bootstrap required, opened a new log
I20250904 07:53:44.456290 5478 ts_tablet_manager.cc:1397] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:44.456804 5478 raft_consensus.cc:357] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.456979 5478 raft_consensus.cc:383] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:44.457041 5478 raft_consensus.cc:738] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.457228 5478 consensus_queue.cc:260] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.457378 5478 raft_consensus.cc:397] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.457439 5478 raft_consensus.cc:491] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.457508 5478 raft_consensus.cc:3058] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:44.459235 5478 raft_consensus.cc:513] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.459383 5478 leader_election.cc:304] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e76acc3b69541108fadf91060ca82b2; no voters:
I20250904 07:53:44.459676 5478 leader_election.cc:290] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:44.459765 5480 raft_consensus.cc:2802] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:44.460179 5480 raft_consensus.cc:695] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Running, Role: LEADER
I20250904 07:53:44.460377 5480 consensus_queue.cc:237] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.460448 5478 ts_tablet_manager.cc:1428] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:44.460644 5478 tablet_bootstrap.cc:492] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: Bootstrap starting.
I20250904 07:53:44.462103 5478 tablet_bootstrap.cc:654] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:44.462134 5323 catalog_manager.cc:5582] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e76acc3b69541108fadf91060ca82b2 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7e76acc3b69541108fadf91060ca82b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:44.463557 5478 tablet_bootstrap.cc:492] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: No bootstrap required, opened a new log
I20250904 07:53:44.463692 5478 ts_tablet_manager.cc:1397] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:44.464264 5478 raft_consensus.cc:357] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.464380 5478 raft_consensus.cc:383] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:44.464412 5478 raft_consensus.cc:738] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.464589 5478 consensus_queue.cc:260] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.464692 5478 raft_consensus.cc:397] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.464751 5478 raft_consensus.cc:491] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.464819 5478 raft_consensus.cc:3058] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:44.466446 5478 raft_consensus.cc:513] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.466619 5478 leader_election.cc:304] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e76acc3b69541108fadf91060ca82b2; no voters:
I20250904 07:53:44.466771 5478 leader_election.cc:290] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:44.466828 5480 raft_consensus.cc:2802] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:44.466951 5480 raft_consensus.cc:695] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Running, Role: LEADER
I20250904 07:53:44.467192 5478 ts_tablet_manager.cc:1428] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20250904 07:53:44.467123 5480 consensus_queue.cc:237] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.469323 5323 catalog_manager.cc:5582] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e76acc3b69541108fadf91060ca82b2 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7e76acc3b69541108fadf91060ca82b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:44.487264 5323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57272:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250904 07:53:44.492802 5429 tablet_service.cc:1468] Processing CreateTablet for tablet 21be4bb670d144c88624f3421fb13333 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9262b7f4b3664c1b8404710ed664273e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250904 07:53:44.493237 5429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21be4bb670d144c88624f3421fb13333. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:44.497962 5478 tablet_bootstrap.cc:492] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: Bootstrap starting.
I20250904 07:53:44.499047 5478 tablet_bootstrap.cc:654] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:44.500288 5478 tablet_bootstrap.cc:492] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: No bootstrap required, opened a new log
I20250904 07:53:44.500383 5478 ts_tablet_manager.cc:1397] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:44.500921 5478 raft_consensus.cc:357] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.501048 5478 raft_consensus.cc:383] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:44.501089 5478 raft_consensus.cc:738] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Initialized, Role: FOLLOWER
I20250904 07:53:44.501319 5478 consensus_queue.cc:260] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.501469 5478 raft_consensus.cc:397] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:44.501541 5478 raft_consensus.cc:491] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:44.501621 5478 raft_consensus.cc:3058] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:44.503537 5478 raft_consensus.cc:513] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.503713 5478 leader_election.cc:304] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e76acc3b69541108fadf91060ca82b2; no voters:
I20250904 07:53:44.503859 5478 leader_election.cc:290] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:44.503926 5480 raft_consensus.cc:2802] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:44.504051 5480 raft_consensus.cc:695] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e76acc3b69541108fadf91060ca82b2, State: Running, Role: LEADER
I20250904 07:53:44.504302 5478 ts_tablet_manager.cc:1428] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:44.504238 5480 consensus_queue.cc:237] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } }
I20250904 07:53:44.504887 5482 tablet_replica.cc:440] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: TxnStatusTablet state changed. Reason: New leader 7e76acc3b69541108fadf91060ca82b2. Latest consensus state: current_term: 1 leader_uuid: "7e76acc3b69541108fadf91060ca82b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } } }
I20250904 07:53:44.504958 5481 tablet_replica.cc:440] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e76acc3b69541108fadf91060ca82b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } } }
I20250904 07:53:44.505048 5482 tablet_replica.cc:443] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: This TxnStatusTablet replica's current role is: LEADER
I20250904 07:53:44.505087 5481 tablet_replica.cc:443] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: This TxnStatusTablet replica's current role is: LEADER
I20250904 07:53:44.505965 5494 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250904 07:53:44.506211 5494 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250904 07:53:44.506328 5322 catalog_manager.cc:5582] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e76acc3b69541108fadf91060ca82b2 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7e76acc3b69541108fadf91060ca82b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e76acc3b69541108fadf91060ca82b2" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40227 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:44.970542 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:44.982453 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:44.983049 413 tablet_replica.cc:331] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2: stopping tablet replica
I20250904 07:53:44.983253 413 raft_consensus.cc:2241] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.983434 413 raft_consensus.cc:2270] T 21be4bb670d144c88624f3421fb13333 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:44.984372 413 tablet_replica.cc:331] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2: stopping tablet replica
I20250904 07:53:44.984508 413 raft_consensus.cc:2241] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.984645 413 raft_consensus.cc:2270] T adf8a17f8cd649f9b11dfd8386e5f471 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:44.985669 413 tablet_replica.cc:331] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2: stopping tablet replica
I20250904 07:53:44.985813 413 raft_consensus.cc:2241] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:44.985955 413 raft_consensus.cc:2270] T 0ceb771bee8541c2904d3b171a6bf6b8 P 7e76acc3b69541108fadf91060ca82b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:45.002094 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:45.007627 413 master.cc:561] Master@127.0.103.126:45937 shutting down...
I20250904 07:53:45.017940 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:45.018154 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:45.018256 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 56218c64c21149118af22213d38cf65e: stopping tablet replica
I20250904 07:53:45.031692 413 master.cc:583] Master@127.0.103.126:45937 shutdown complete.
[ OK ] ClientTest.FlushTxnSessionsOnCommit (729 ms)
[ RUN ] ClientTest.TxnCorruptedToken
I20250904 07:53:45.046264 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:42649
I20250904 07:53:45.046801 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:45.049167 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.049222 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.049960 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:45.054134 413 server_base.cc:1047] running on GCE node
I20250904 07:53:45.054399 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:45.054461 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:45.054497 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972425054494 us; error 0 us; skew 500 ppm
I20250904 07:53:45.054651 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:45.055932 413 webserver.cc:480] Webserver started at http://127.0.103.126:34393/ using document root <none> and password file <none>
I20250904 07:53:45.056152 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:45.056221 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:45.056334 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:45.057034 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "329dd13a29aa4f76b759c47fd6b0c241"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.060027 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:45.061758 5513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.062104 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:45.062196 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "329dd13a29aa4f76b759c47fd6b0c241"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.062289 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:45.074214 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:45.074643 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:45.090615 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:42649
I20250904 07:53:45.090617 5574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:42649 every 8 connection(s)
I20250904 07:53:45.091976 5575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.097999 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241: Bootstrap starting.
I20250904 07:53:45.099439 5575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.100987 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241: No bootstrap required, opened a new log
I20250904 07:53:45.101681 5575 raft_consensus.cc:357] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER }
I20250904 07:53:45.101828 5575 raft_consensus.cc:383] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.101886 5575 raft_consensus.cc:738] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329dd13a29aa4f76b759c47fd6b0c241, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.102093 5575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER }
I20250904 07:53:45.102233 5575 raft_consensus.cc:397] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.102293 5575 raft_consensus.cc:491] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.102365 5575 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.104965 5575 raft_consensus.cc:513] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER }
I20250904 07:53:45.105161 5575 leader_election.cc:304] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 329dd13a29aa4f76b759c47fd6b0c241; no voters:
I20250904 07:53:45.105576 5575 leader_election.cc:290] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.105731 5578 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.106396 5578 raft_consensus.cc:695] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 1 LEADER]: Becoming Leader. State: Replica: 329dd13a29aa4f76b759c47fd6b0c241, State: Running, Role: LEADER
I20250904 07:53:45.106424 5575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:45.106632 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER }
I20250904 07:53:45.107323 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329dd13a29aa4f76b759c47fd6b0c241. Latest consensus state: current_term: 1 leader_uuid: "329dd13a29aa4f76b759c47fd6b0c241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER } }
I20250904 07:53:45.107409 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "329dd13a29aa4f76b759c47fd6b0c241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "329dd13a29aa4f76b759c47fd6b0c241" member_type: VOTER } }
I20250904 07:53:45.107493 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:45.107481 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:45.117435 5582 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:45.118749 5582 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:45.120741 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:45.121773 5582 catalog_manager.cc:1349] Generated new cluster ID: 34672e1e74e741de88e0dcaa99ccc028
I20250904 07:53:45.121850 5582 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:45.134651 5582 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:45.135394 5582 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:45.149588 5582 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241: Generated new TSK 0
I20250904 07:53:45.149775 5582 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:45.153298 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:45.155769 5596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.156203 5597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.157238 5599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:45.157549 413 server_base.cc:1047] running on GCE node
I20250904 07:53:45.157788 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:45.157842 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:45.157867 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972425157864 us; error 0 us; skew 500 ppm
I20250904 07:53:45.158004 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:45.159174 413 webserver.cc:480] Webserver started at http://127.0.103.65:36339/ using document root <none> and password file <none>
I20250904 07:53:45.159354 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:45.159411 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:45.159525 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:45.160084 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "e984dea29c7d457cab88baf1d43f8ba8"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.162338 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:53:45.163767 5604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.164156 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:45.164247 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "e984dea29c7d457cab88baf1d43f8ba8"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.164355 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTest.TxnCorruptedToken.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:45.175617 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:45.176013 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:45.183838 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:45.183941 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.184044 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:45.184103 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.213948 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41373
I20250904 07:53:45.213970 5674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41373 every 8 connection(s)
I20250904 07:53:45.215516 5675 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42649
I20250904 07:53:45.215643 5675 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:45.216004 5675 heartbeater.cc:507] Master 127.0.103.126:42649 requested a full tablet report, sending...
I20250904 07:53:45.216784 5530 ts_manager.cc:194] Registered new tserver with Master: e984dea29c7d457cab88baf1d43f8ba8 (127.0.103.65:41373)
I20250904 07:53:45.217207 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002296922s
I20250904 07:53:45.218225 5530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49706
I20250904 07:53:45.229539 5675 heartbeater.cc:499] Master 127.0.103.126:42649 was elected leader, sending a full tablet report...
I20250904 07:53:45.232010 5529 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49728:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:45.245398 5636 tablet_service.cc:1468] Processing CreateTablet for tablet 6feba5b6b66a4df3ab30c7b377ae3eaf (DEFAULT_TABLE table=client-testtb [id=843d0ea02c564564bcb1d0ceedb9ede9]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:45.245399 5638 tablet_service.cc:1468] Processing CreateTablet for tablet f1e5af8315af4a23ab4feda4ab2575b1 (DEFAULT_TABLE table=client-testtb [id=843d0ea02c564564bcb1d0ceedb9ede9]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:45.245800 5636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6feba5b6b66a4df3ab30c7b377ae3eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.246034 5638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e5af8315af4a23ab4feda4ab2575b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.250605 5685 tablet_bootstrap.cc:492] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: Bootstrap starting.
I20250904 07:53:45.251785 5685 tablet_bootstrap.cc:654] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.253388 5685 tablet_bootstrap.cc:492] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: No bootstrap required, opened a new log
I20250904 07:53:45.253507 5685 ts_tablet_manager.cc:1397] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.254079 5685 raft_consensus.cc:357] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.254195 5685 raft_consensus.cc:383] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.254226 5685 raft_consensus.cc:738] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e984dea29c7d457cab88baf1d43f8ba8, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.254385 5685 consensus_queue.cc:260] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.254488 5685 raft_consensus.cc:397] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.254529 5685 raft_consensus.cc:491] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.254577 5685 raft_consensus.cc:3058] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.256287 5685 raft_consensus.cc:513] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.256459 5685 leader_election.cc:304] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e984dea29c7d457cab88baf1d43f8ba8; no voters:
I20250904 07:53:45.256940 5685 leader_election.cc:290] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.257006 5687 raft_consensus.cc:2802] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.257625 5685 ts_tablet_manager.cc:1428] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250904 07:53:45.257925 5685 tablet_bootstrap.cc:492] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: Bootstrap starting.
I20250904 07:53:45.259732 5685 tablet_bootstrap.cc:654] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.261801 5685 tablet_bootstrap.cc:492] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: No bootstrap required, opened a new log
I20250904 07:53:45.261958 5685 ts_tablet_manager.cc:1397] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:53:45.262818 5685 raft_consensus.cc:357] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.263073 5685 raft_consensus.cc:383] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.263139 5685 raft_consensus.cc:738] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e984dea29c7d457cab88baf1d43f8ba8, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.263394 5685 consensus_queue.cc:260] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.263566 5685 raft_consensus.cc:397] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.263630 5685 raft_consensus.cc:491] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.263638 5687 raft_consensus.cc:695] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 1 LEADER]: Becoming Leader. State: Replica: e984dea29c7d457cab88baf1d43f8ba8, State: Running, Role: LEADER
I20250904 07:53:45.263710 5685 raft_consensus.cc:3058] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.263854 5687 consensus_queue.cc:237] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.266136 5530 catalog_manager.cc:5582] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e984dea29c7d457cab88baf1d43f8ba8 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e984dea29c7d457cab88baf1d43f8ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.266794 5685 raft_consensus.cc:513] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.267693 5685 leader_election.cc:304] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e984dea29c7d457cab88baf1d43f8ba8; no voters:
I20250904 07:53:45.267927 5685 leader_election.cc:290] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.269179 5688 raft_consensus.cc:2802] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.269589 5688 raft_consensus.cc:695] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 1 LEADER]: Becoming Leader. State: Replica: e984dea29c7d457cab88baf1d43f8ba8, State: Running, Role: LEADER
I20250904 07:53:45.269779 5688 consensus_queue.cc:237] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } }
I20250904 07:53:45.270691 5685 ts_tablet_manager.cc:1428] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20250904 07:53:45.271756 5530 catalog_manager.cc:5582] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e984dea29c7d457cab88baf1d43f8ba8 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e984dea29c7d457cab88baf1d43f8ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e984dea29c7d457cab88baf1d43f8ba8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41373 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.285352 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:45.295519 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:45.295800 413 tablet_replica.cc:331] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8: stopping tablet replica
I20250904 07:53:45.296062 413 raft_consensus.cc:2241] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:45.296234 413 raft_consensus.cc:2270] T f1e5af8315af4a23ab4feda4ab2575b1 P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:45.297132 413 tablet_replica.cc:331] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8: stopping tablet replica
I20250904 07:53:45.297251 413 raft_consensus.cc:2241] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:45.297389 413 raft_consensus.cc:2270] T 6feba5b6b66a4df3ab30c7b377ae3eaf P e984dea29c7d457cab88baf1d43f8ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:45.312014 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:45.317055 413 master.cc:561] Master@127.0.103.126:42649 shutting down...
I20250904 07:53:45.325402 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:45.325605 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:45.325739 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 329dd13a29aa4f76b759c47fd6b0c241: stopping tablet replica
I20250904 07:53:45.339445 413 master.cc:583] Master@127.0.103.126:42649 shutdown complete.
[ OK ] ClientTest.TxnCorruptedToken (303 ms)
[----------] 20 tests from ClientTest (52212 ms total)
[----------] 1 test from TableKeyRangeTest
[ RUN ] TableKeyRangeTest.TestGetTableKeyRange
I20250904 07:53:45.348548 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39499
I20250904 07:53:45.349035 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:45.350992 5695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.351064 5696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.351610 5698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:45.351948 413 server_base.cc:1047] running on GCE node
I20250904 07:53:45.352169 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:45.352229 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:45.352272 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972425352267 us; error 0 us; skew 500 ppm
I20250904 07:53:45.352423 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:45.353502 413 webserver.cc:480] Webserver started at http://127.0.103.126:43277/ using document root <none> and password file <none>
I20250904 07:53:45.353681 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:45.353739 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:45.353852 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:45.354439 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "42a39daac14f42bf8447a282616c2ae7"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.356698 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:45.358199 5703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.358562 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:45.358645 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "42a39daac14f42bf8447a282616c2ae7"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.358724 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:45.374531 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:45.374922 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:45.385988 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39499
I20250904 07:53:45.386009 5764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39499 every 8 connection(s)
I20250904 07:53:45.387068 5765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.390967 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7: Bootstrap starting.
I20250904 07:53:45.392019 5765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.393347 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7: No bootstrap required, opened a new log
I20250904 07:53:45.393858 5765 raft_consensus.cc:357] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER }
I20250904 07:53:45.393965 5765 raft_consensus.cc:383] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.394013 5765 raft_consensus.cc:738] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a39daac14f42bf8447a282616c2ae7, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.394213 5765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER }
I20250904 07:53:45.394317 5765 raft_consensus.cc:397] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.394378 5765 raft_consensus.cc:491] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.394455 5765 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.396138 5765 raft_consensus.cc:513] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER }
I20250904 07:53:45.396308 5765 leader_election.cc:304] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42a39daac14f42bf8447a282616c2ae7; no voters:
I20250904 07:53:45.396668 5765 leader_election.cc:290] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.396771 5768 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.397226 5768 raft_consensus.cc:695] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 42a39daac14f42bf8447a282616c2ae7, State: Running, Role: LEADER
I20250904 07:53:45.397428 5768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER }
I20250904 07:53:45.397480 5765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:45.398156 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42a39daac14f42bf8447a282616c2ae7. Latest consensus state: current_term: 1 leader_uuid: "42a39daac14f42bf8447a282616c2ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER } }
I20250904 07:53:45.398293 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:45.398644 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42a39daac14f42bf8447a282616c2ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a39daac14f42bf8447a282616c2ae7" member_type: VOTER } }
I20250904 07:53:45.398861 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:45.399235 5773 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:45.400403 5773 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:45.402762 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:45.402871 5773 catalog_manager.cc:1349] Generated new cluster ID: 7ab11dfe972b4848b6f48249adc3e279
I20250904 07:53:45.402959 5773 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:45.409477 5773 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:45.410207 5773 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:45.416586 5773 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7: Generated new TSK 0
I20250904 07:53:45.416767 5773 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:45.418948 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:45.421201 5787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:45.421207 5786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:45.422552 413 server_base.cc:1047] running on GCE node
W20250904 07:53:45.422566 5789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:45.422892 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:45.422951 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:45.422977 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972425422975 us; error 0 us; skew 500 ppm
I20250904 07:53:45.423116 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:45.424167 413 webserver.cc:480] Webserver started at http://127.0.103.65:32965/ using document root <none> and password file <none>
I20250904 07:53:45.424361 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:45.424420 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:45.424536 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:45.425168 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "7f7bbc295d4d40c3b6c4538c67415db6"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.427364 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:45.428725 5794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.429042 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:45.429152 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "7f7bbc295d4d40c3b6c4538c67415db6"
format_stamp: "Formatted at 2025-09-04 07:53:45 on dist-test-slave-ghlz"
I20250904 07:53:45.429279 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.TableKeyRangeTest.TestGetTableKeyRange.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:45.450498 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:45.450918 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:45.452410 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:45.452494 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.452572 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:45.452629 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:45.472004 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40721
I20250904 07:53:45.472030 5864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40721 every 8 connection(s)
I20250904 07:53:45.473572 5865 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39499
I20250904 07:53:45.473670 5865 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:45.473930 5865 heartbeater.cc:507] Master 127.0.103.126:39499 requested a full tablet report, sending...
I20250904 07:53:45.474578 5720 ts_manager.cc:194] Registered new tserver with Master: 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65:40721)
I20250904 07:53:45.475247 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002272285s
I20250904 07:53:45.475530 5720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36150
I20250904 07:53:45.484400 5720 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36162:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:45.486665 5865 heartbeater.cc:499] Master 127.0.103.126:39499 was elected leader, sending a full tablet report...
I20250904 07:53:45.495900 5824 tablet_service.cc:1468] Processing CreateTablet for tablet 49037ece9458400b8233b2591bb85aec (DEFAULT_TABLE table=client-testtb [id=bdcc091fb0ba42cdb8083232db4432f6]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:53:45.495901 5825 tablet_service.cc:1468] Processing CreateTablet for tablet 9bf4702c3a6f4ff7b9d903280e7a9c58 (DEFAULT_TABLE table=client-testtb [id=bdcc091fb0ba42cdb8083232db4432f6]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:53:45.496371 5825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf4702c3a6f4ff7b9d903280e7a9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.496663 5824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49037ece9458400b8233b2591bb85aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.501499 5875 tablet_bootstrap.cc:492] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.502499 5875 tablet_bootstrap.cc:654] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.503705 5875 tablet_bootstrap.cc:492] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.503798 5875 ts_tablet_manager.cc:1397] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:45.504308 5875 raft_consensus.cc:357] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.504426 5875 raft_consensus.cc:383] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.504491 5875 raft_consensus.cc:738] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.504660 5875 consensus_queue.cc:260] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.504766 5875 raft_consensus.cc:397] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.504825 5875 raft_consensus.cc:491] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.504920 5875 raft_consensus.cc:3058] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.506557 5875 raft_consensus.cc:513] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.506740 5875 leader_election.cc:304] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.507083 5875 leader_election.cc:290] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.507223 5877 raft_consensus.cc:2802] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.507628 5877 raft_consensus.cc:695] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.507691 5875 ts_tablet_manager.cc:1428] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:45.507922 5875 tablet_bootstrap.cc:492] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.507869 5877 consensus_queue.cc:237] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.509151 5875 tablet_bootstrap.cc:654] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.510018 5720 catalog_manager.cc:5582] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.510486 5875 tablet_bootstrap.cc:492] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.510603 5875 ts_tablet_manager.cc:1397] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.511217 5875 raft_consensus.cc:357] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.511348 5875 raft_consensus.cc:383] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.511407 5875 raft_consensus.cc:738] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.511610 5875 consensus_queue.cc:260] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.511744 5875 raft_consensus.cc:397] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.511806 5875 raft_consensus.cc:491] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.511873 5875 raft_consensus.cc:3058] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.514508 5875 raft_consensus.cc:513] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.514684 5875 leader_election.cc:304] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.514833 5875 leader_election.cc:290] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.514911 5877 raft_consensus.cc:2802] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.515130 5875 ts_tablet_manager.cc:1428] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:45.515249 5877 raft_consensus.cc:695] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.515440 5877 consensus_queue.cc:237] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.516973 5720 catalog_manager.cc:5582] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.523532 5720 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36162:
name: "client-testrange"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\007\001\000\372\000\000\000\006\001\000\372\000\000\000\007\001\000\364\001\000\000\006\001\000\364\001\000\000\007\001\000\356\002\000\000\006\001\000\356\002\000\000\007\001\000\350\003\000\000""\006\001\000\000\000\000\000\007\001\000\372\000\000\000\006\001\000\372\000\000\000\007\001\000\364\001\000\000\006\001\000\364\001\000\000\007\001\000\356\002\000\000\006\001\000\356\002\000\000\007\001\000\350\003\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:53:45.529970 5824 tablet_service.cc:1468] Processing CreateTablet for tablet 2b308873b8c94c73b5149f2fb3167bac (DEFAULT_TABLE table=client-testrange [id=f1c33e5fbe644f75a2357230655ebd91]), partition=RANGE (key) PARTITION 0 <= VALUES < 250
I20250904 07:53:45.530318 5824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b308873b8c94c73b5149f2fb3167bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.530251 5825 tablet_service.cc:1468] Processing CreateTablet for tablet ae908956c3f44091a31ec4b1f883bf80 (DEFAULT_TABLE table=client-testrange [id=f1c33e5fbe644f75a2357230655ebd91]), partition=RANGE (key) PARTITION 250 <= VALUES < 500
I20250904 07:53:45.530380 5823 tablet_service.cc:1468] Processing CreateTablet for tablet 0f917ec816764b2fa60109acf43a3608 (DEFAULT_TABLE table=client-testrange [id=f1c33e5fbe644f75a2357230655ebd91]), partition=RANGE (key) PARTITION 500 <= VALUES < 750
I20250904 07:53:45.530714 5825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae908956c3f44091a31ec4b1f883bf80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.530861 5822 tablet_service.cc:1468] Processing CreateTablet for tablet 9c510a21310e4389b1068386f4e5676f (DEFAULT_TABLE table=client-testrange [id=f1c33e5fbe644f75a2357230655ebd91]), partition=RANGE (key) PARTITION 750 <= VALUES < 1000
I20250904 07:53:45.531241 5822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c510a21310e4389b1068386f4e5676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.533054 5823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f917ec816764b2fa60109acf43a3608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:45.536252 5875 tablet_bootstrap.cc:492] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.537470 5875 tablet_bootstrap.cc:654] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.538815 5875 tablet_bootstrap.cc:492] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.538913 5875 ts_tablet_manager.cc:1397] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.539419 5875 raft_consensus.cc:357] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.539528 5875 raft_consensus.cc:383] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.539577 5875 raft_consensus.cc:738] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.539753 5875 consensus_queue.cc:260] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.539849 5875 raft_consensus.cc:397] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.539912 5875 raft_consensus.cc:491] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.539985 5875 raft_consensus.cc:3058] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.541955 5875 raft_consensus.cc:513] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.542130 5875 leader_election.cc:304] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.542285 5875 leader_election.cc:290] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.542372 5877 raft_consensus.cc:2802] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.542518 5877 raft_consensus.cc:695] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.542694 5875 ts_tablet_manager.cc:1428] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:45.542694 5877 consensus_queue.cc:237] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.542964 5875 tablet_bootstrap.cc:492] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.544096 5875 tablet_bootstrap.cc:654] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.545943 5875 tablet_bootstrap.cc:492] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.546072 5875 ts_tablet_manager.cc:1397] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.546706 5875 raft_consensus.cc:357] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.546846 5875 raft_consensus.cc:383] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.546892 5875 raft_consensus.cc:738] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.546768 5720 catalog_manager.cc:5582] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.547083 5875 consensus_queue.cc:260] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.547224 5875 raft_consensus.cc:397] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.547289 5875 raft_consensus.cc:491] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.547358 5875 raft_consensus.cc:3058] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.549533 5875 raft_consensus.cc:513] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.549716 5875 leader_election.cc:304] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.549857 5875 leader_election.cc:290] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.549952 5877 raft_consensus.cc:2802] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.550112 5877 raft_consensus.cc:695] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.550239 5875 ts_tablet_manager.cc:1428] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:53:45.550299 5877 consensus_queue.cc:237] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.550451 5875 tablet_bootstrap.cc:492] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.551674 5875 tablet_bootstrap.cc:654] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.552069 5720 catalog_manager.cc:5582] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.552997 5875 tablet_bootstrap.cc:492] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.553112 5875 ts_tablet_manager.cc:1397] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.553648 5875 raft_consensus.cc:357] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.553793 5875 raft_consensus.cc:383] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.553854 5875 raft_consensus.cc:738] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.554029 5875 consensus_queue.cc:260] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.554173 5875 raft_consensus.cc:397] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.554241 5875 raft_consensus.cc:491] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.554301 5875 raft_consensus.cc:3058] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.556368 5875 raft_consensus.cc:513] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.556536 5875 leader_election.cc:304] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.556679 5875 leader_election.cc:290] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.556756 5877 raft_consensus.cc:2802] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.556947 5877 raft_consensus.cc:695] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.557090 5875 ts_tablet_manager.cc:1428] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250904 07:53:45.557142 5877 consensus_queue.cc:237] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.557319 5875 tablet_bootstrap.cc:492] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: Bootstrap starting.
I20250904 07:53:45.558482 5875 tablet_bootstrap.cc:654] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:45.558843 5719 catalog_manager.cc:5582] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.559966 5875 tablet_bootstrap.cc:492] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: No bootstrap required, opened a new log
I20250904 07:53:45.560084 5875 ts_tablet_manager.cc:1397] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:45.560632 5875 raft_consensus.cc:357] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.560767 5875 raft_consensus.cc:383] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:45.560829 5875 raft_consensus.cc:738] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Initialized, Role: FOLLOWER
I20250904 07:53:45.561028 5875 consensus_queue.cc:260] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.561187 5875 raft_consensus.cc:397] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:45.561261 5875 raft_consensus.cc:491] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:45.561309 5875 raft_consensus.cc:3058] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:45.563567 5875 raft_consensus.cc:513] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.563725 5875 leader_election.cc:304] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7bbc295d4d40c3b6c4538c67415db6; no voters:
I20250904 07:53:45.563868 5875 leader_election.cc:290] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:45.563922 5877 raft_consensus.cc:2802] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:45.564081 5877 raft_consensus.cc:695] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7bbc295d4d40c3b6c4538c67415db6, State: Running, Role: LEADER
I20250904 07:53:45.564254 5875 ts_tablet_manager.cc:1428] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250904 07:53:45.564272 5877 consensus_queue.cc:237] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } }
I20250904 07:53:45.566051 5719 catalog_manager.cc:5582] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7bbc295d4d40c3b6c4538c67415db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7bbc295d4d40c3b6c4538c67415db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40721 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:45.626134 5709 maintenance_manager.cc:419] P 42a39daac14f42bf8447a282616c2ae7: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20250904 07:53:45.657156 5708 maintenance_manager.cc:643] P 42a39daac14f42bf8447a282616c2ae7: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.022s sys 0.008s Metrics: {"bytes_written":8087,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":630,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":11}
I20250904 07:53:45.658372 5709 maintenance_manager.cc:419] P 42a39daac14f42bf8447a282616c2ae7: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 646 bytes on disk
I20250904 07:53:45.659188 5708 maintenance_manager.cc:643] P 42a39daac14f42bf8447a282616c2ae7: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20250904 07:53:45.723634 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling FlushMRSOp(0f917ec816764b2fa60109acf43a3608): perf score=1.000000
I20250904 07:53:45.786239 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: FlushMRSOp(0f917ec816764b2fa60109acf43a3608) complete. Timing: real 0.062s user 0.025s sys 0.004s Metrics: {"bytes_written":4816,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":710,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250}
I20250904 07:53:45.788911 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling FlushMRSOp(2b308873b8c94c73b5149f2fb3167bac): perf score=1.000000
I20250904 07:53:45.818629 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: FlushMRSOp(2b308873b8c94c73b5149f2fb3167bac) complete. Timing: real 0.030s user 0.016s sys 0.005s Metrics: {"bytes_written":4799,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1191,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250,"spinlock_wait_cycles":266112}
I20250904 07:53:45.819470 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling UndoDeltaBlockGCOp(2b308873b8c94c73b5149f2fb3167bac): 1705 bytes on disk
I20250904 07:53:45.819972 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: UndoDeltaBlockGCOp(2b308873b8c94c73b5149f2fb3167bac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250904 07:53:45.820550 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling FlushMRSOp(9c510a21310e4389b1068386f4e5676f): perf score=1.000000
I20250904 07:53:45.865692 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: FlushMRSOp(9c510a21310e4389b1068386f4e5676f) complete. Timing: real 0.045s user 0.035s sys 0.004s Metrics: {"bytes_written":4826,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1197,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250}
I20250904 07:53:45.867697 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling UndoDeltaBlockGCOp(9c510a21310e4389b1068386f4e5676f): 1705 bytes on disk
I20250904 07:53:45.868269 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: UndoDeltaBlockGCOp(9c510a21310e4389b1068386f4e5676f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20250904 07:53:45.868794 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling FlushMRSOp(ae908956c3f44091a31ec4b1f883bf80): perf score=1.000000
I20250904 07:53:45.896716 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: FlushMRSOp(ae908956c3f44091a31ec4b1f883bf80) complete. Timing: real 0.028s user 0.024s sys 0.000s Metrics: {"bytes_written":4818,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1083,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":896,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250}
I20250904 07:53:45.897694 5866 maintenance_manager.cc:419] P 7f7bbc295d4d40c3b6c4538c67415db6: Scheduling UndoDeltaBlockGCOp(ae908956c3f44091a31ec4b1f883bf80): 1705 bytes on disk
I20250904 07:53:45.898348 5799 maintenance_manager.cc:643] P 7f7bbc295d4d40c3b6c4538c67415db6: UndoDeltaBlockGCOp(ae908956c3f44091a31ec4b1f883bf80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20250904 07:53:46.350324 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:46.360459 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:46.360671 413 tablet_replica.cc:331] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.360888 413 raft_consensus.cc:2241] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.361094 413 raft_consensus.cc:2270] T 9c510a21310e4389b1068386f4e5676f P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.361965 413 tablet_replica.cc:331] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.362107 413 raft_consensus.cc:2241] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.362279 413 raft_consensus.cc:2270] T 2b308873b8c94c73b5149f2fb3167bac P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.362937 413 tablet_replica.cc:331] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.363049 413 raft_consensus.cc:2241] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.363186 413 raft_consensus.cc:2270] T ae908956c3f44091a31ec4b1f883bf80 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.363766 413 tablet_replica.cc:331] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.363885 413 raft_consensus.cc:2241] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.364012 413 raft_consensus.cc:2270] T 49037ece9458400b8233b2591bb85aec P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.364529 413 tablet_replica.cc:331] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.364640 413 raft_consensus.cc:2241] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.364789 413 raft_consensus.cc:2270] T 0f917ec816764b2fa60109acf43a3608 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.365447 413 tablet_replica.cc:331] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6: stopping tablet replica
I20250904 07:53:46.365551 413 raft_consensus.cc:2241] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.365672 413 raft_consensus.cc:2270] T 9bf4702c3a6f4ff7b9d903280e7a9c58 P 7f7bbc295d4d40c3b6c4538c67415db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.380223 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:46.388170 413 master.cc:561] Master@127.0.103.126:39499 shutting down...
I20250904 07:53:46.405887 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:46.408315 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:46.408478 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 42a39daac14f42bf8447a282616c2ae7: stopping tablet replica
I20250904 07:53:46.431059 413 master.cc:583] Master@127.0.103.126:39499 shutdown complete.
[ OK ] TableKeyRangeTest.TestGetTableKeyRange (1093 ms)
[----------] 1 test from TableKeyRangeTest (1093 ms total)
[----------] 1 test from MultiTServerClientTest
[ RUN ] MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers
I20250904 07:53:46.442179 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39793
I20250904 07:53:46.442620 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:46.444695 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.444741 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.445087 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.446031 413 server_base.cc:1047] running on GCE node
I20250904 07:53:46.446239 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:46.446306 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:46.446360 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972426446339 us; error 0 us; skew 500 ppm
I20250904 07:53:46.446532 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:46.447619 413 webserver.cc:480] Webserver started at http://127.0.103.126:46877/ using document root <none> and password file <none>
I20250904 07:53:46.447821 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:46.447922 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:46.448062 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:46.448629 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.450922 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:46.452296 6052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.452631 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:46.452726 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.452875 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:46.467375 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:46.467829 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:46.481024 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39793
I20250904 07:53:46.481036 6103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39793 every 8 connection(s)
I20250904 07:53:46.482402 6104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.488258 6104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7: Bootstrap starting.
I20250904 07:53:46.489619 6104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.491178 6104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7: No bootstrap required, opened a new log
I20250904 07:53:46.491906 6104 raft_consensus.cc:357] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER }
I20250904 07:53:46.492070 6104 raft_consensus.cc:383] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.492154 6104 raft_consensus.cc:738] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea8ca3f3ef4e73955b5e5408a9bca7, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.492375 6104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER }
I20250904 07:53:46.492539 6104 raft_consensus.cc:397] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:53:46.492621 6104 raft_consensus.cc:491] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:53:46.492715 6104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:46.495332 6104 raft_consensus.cc:513] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER }
I20250904 07:53:46.495559 6104 leader_election.cc:304] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ea8ca3f3ef4e73955b5e5408a9bca7; no voters:
I20250904 07:53:46.495958 6104 leader_election.cc:290] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:53:46.496023 6107 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:46.496693 6107 raft_consensus.cc:695] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 1 LEADER]: Becoming Leader. State: Replica: 26ea8ca3f3ef4e73955b5e5408a9bca7, State: Running, Role: LEADER
I20250904 07:53:46.496754 6104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:53:46.496953 6107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER }
I20250904 07:53:46.497707 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER } }
I20250904 07:53:46.497864 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:46.498243 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ea8ca3f3ef4e73955b5e5408a9bca7. Latest consensus state: current_term: 1 leader_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea8ca3f3ef4e73955b5e5408a9bca7" member_type: VOTER } }
I20250904 07:53:46.498410 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [sys.catalog]: This master's current role is: LEADER
I20250904 07:53:46.501111 6112 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:53:46.502102 6112 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:53:46.505594 6112 catalog_manager.cc:1349] Generated new cluster ID: 337dfcca0bd04a418c758e4d6dd346c4
I20250904 07:53:46.505683 6112 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:53:46.527748 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:53:46.528239 6112 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:53:46.529448 6112 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:53:46.541643 6112 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7: Generated new TSK 0
I20250904 07:53:46.541896 6112 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:53:46.545697 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:46.548183 6126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.549095 6127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.553637 6130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.556542 413 server_base.cc:1047] running on GCE node
I20250904 07:53:46.556782 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:46.556872 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:46.556936 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972426556915 us; error 0 us; skew 500 ppm
I20250904 07:53:46.557112 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:46.558372 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:46.558597 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:46.558677 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:46.558820 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:46.559422 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.561852 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:53:46.563385 6134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.563778 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:46.563879 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.564004 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:46.592212 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:46.592716 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:46.593307 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:46.594210 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:46.594269 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.594360 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:46.594403 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.609144 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:46.609144 6196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:53:46.610726 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:46.624468 6201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.625068 6202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.630743 6197 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:46.630885 6197 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:46.631268 6197 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:46.632305 6069 ts_manager.cc:194] Registered new tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:46.634238 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35824
W20250904 07:53:46.635315 6204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.635773 413 server_base.cc:1047] running on GCE node
I20250904 07:53:46.636057 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:46.636112 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:46.636157 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972426636152 us; error 0 us; skew 500 ppm
I20250904 07:53:46.636320 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:46.637550 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:53:46.637754 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:46.637809 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:46.637921 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:46.638451 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/instance:
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.640637 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:53:46.642117 6209 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.642443 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:46.642532 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.642642 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:46.649302 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:46.649703 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:46.650208 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:46.650974 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:46.651046 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.651134 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:46.651194 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.662644 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:53:46.662673 6271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:53:46.664057 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:46.666754 6276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.667356 6275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.667770 6278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.668079 413 server_base.cc:1047] running on GCE node
I20250904 07:53:46.668351 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:46.668437 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:46.668494 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972426668466 us; error 0 us; skew 500 ppm
I20250904 07:53:46.668653 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:46.669615 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:53:46.669817 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:46.669903 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:46.670027 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:46.670567 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/instance:
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.672813 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:53:46.673967 6272 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:46.674103 6272 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:46.674499 6283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.674531 6272 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:46.674832 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:46.674963 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.675097 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:46.675432 6069 ts_manager.cc:194] Registered new tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:53:46.676254 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35836
I20250904 07:53:46.683416 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:46.683797 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:46.684288 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:46.685057 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:46.685132 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.685240 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:46.685295 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.696727 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:53:46.696754 6345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:53:46.698122 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:46.700568 6350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.700883 6352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:46.701680 6349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:46.703677 413 server_base.cc:1047] running on GCE node
I20250904 07:53:46.703891 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:46.703977 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:46.704041 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972426704020 us; error 0 us; skew 500 ppm
I20250904 07:53:46.704173 6346 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:46.704234 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:46.704300 6346 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:46.704636 6346 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:46.705472 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:53:46.705546 6069 ts_manager.cc:194] Registered new tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:46.705677 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:46.705752 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:53:46.705904 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:53:46.706344 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35852
I20250904 07:53:46.706466 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/instance:
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.708570 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:46.709868 6357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.710168 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:46.710299 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:46.710422 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:46.733567 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:46.734035 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:46.734562 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:46.735302 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:53:46.735374 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.735481 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:53:46.735548 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:46.747481 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:53:46.747520 6419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:53:46.753440 6420 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:46.753557 6420 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:46.753926 6420 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:46.754647 6069 ts_manager.cc:194] Registered new tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:46.755038 413 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006751746s
I20250904 07:53:46.755353 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35862
I20250904 07:53:46.764556 6069 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35870:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250904 07:53:46.789601 6162 tablet_service.cc:1468] Processing CreateTablet for tablet d54fed14914b4487805737cebff4717a (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.790148 6162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54fed14914b4487805737cebff4717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.792835 6161 tablet_service.cc:1468] Processing CreateTablet for tablet abd498e731794d59bc5fcf9a0de7a6d3 (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.793126 6237 tablet_service.cc:1468] Processing CreateTablet for tablet abd498e731794d59bc5fcf9a0de7a6d3 (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.793262 6161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd498e731794d59bc5fcf9a0de7a6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.793526 6237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd498e731794d59bc5fcf9a0de7a6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.798676 6385 tablet_service.cc:1468] Processing CreateTablet for tablet d54fed14914b4487805737cebff4717a (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.799077 6385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54fed14914b4487805737cebff4717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.807744 6311 tablet_service.cc:1468] Processing CreateTablet for tablet d54fed14914b4487805737cebff4717a (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.808257 6437 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:46.808262 6311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54fed14914b4487805737cebff4717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.810205 6437 tablet_bootstrap.cc:654] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.816995 6310 tablet_service.cc:1468] Processing CreateTablet for tablet abd498e731794d59bc5fcf9a0de7a6d3 (DEFAULT_TABLE table=client-testtb [id=e7ec7b5e119243009ba67e3375146cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:53:46.817469 6310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd498e731794d59bc5fcf9a0de7a6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:53:46.818396 6438 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:53:46.818738 6437 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: No bootstrap required, opened a new log
I20250904 07:53:46.818915 6437 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250904 07:53:46.819726 6437 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.819954 6437 raft_consensus.cc:383] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.820081 6437 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.820408 6437 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.821007 6439 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:53:46.821488 6197 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:46.821708 6438 tablet_bootstrap.cc:654] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.821612 6441 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:46.823433 6439 tablet_bootstrap.cc:654] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.824437 6441 tablet_bootstrap.cc:654] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.821601 6437 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250904 07:53:46.825137 6437 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:46.826649 6437 tablet_bootstrap.cc:654] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.830736 6438 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: No bootstrap required, opened a new log
I20250904 07:53:46.830926 6438 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20250904 07:53:46.831802 6438 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.832064 6438 raft_consensus.cc:383] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.832167 6438 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.832491 6438 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.835047 6441 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: No bootstrap required, opened a new log
I20250904 07:53:46.835296 6441 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250904 07:53:46.836218 6441 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.836642 6441 raft_consensus.cc:383] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.836791 6441 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.837198 6441 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.838205 6346 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:46.838716 6439 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: No bootstrap required, opened a new log
I20250904 07:53:46.838904 6439 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20250904 07:53:46.839620 6439 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.839877 6439 raft_consensus.cc:383] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.840031 6439 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.840334 6439 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.841037 6441 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250904 07:53:46.841156 6438 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.010s user 0.003s sys 0.008s
I20250904 07:53:46.841322 6441 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:46.841645 6420 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:46.841724 6439 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250904 07:53:46.842664 6437 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: No bootstrap required, opened a new log
I20250904 07:53:46.842809 6437 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20250904 07:53:46.842833 6272 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:46.843418 6437 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.843613 6437 raft_consensus.cc:383] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.843706 6437 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.843756 6441 tablet_bootstrap.cc:654] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Neither blocks nor log segments found. Creating new log.
I20250904 07:53:46.843923 6437 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.844542 6437 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250904 07:53:46.846939 6441 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: No bootstrap required, opened a new log
I20250904 07:53:46.847069 6441 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250904 07:53:46.847524 6441 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.847721 6441 raft_consensus.cc:383] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:53:46.847795 6441 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:46.848024 6441 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:46.848407 6441 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:53:46.975703 6449 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:46.975936 6449 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.976711 6449 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:46.982306 6172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:46.982422 6321 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:46.982517 6172 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 0.
I20250904 07:53:46.982625 6321 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 0.
I20250904 07:53:46.982975 6358 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:46.983335 6449 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:46.983417 6449 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:46.983479 6449 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:46.985428 6449 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.985909 6449 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 1 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:46.986234 6172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:46.986366 6172 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:46.986328 6321 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:46.986517 6321 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:46.988209 6172 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 1.
I20250904 07:53:46.988345 6321 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 1.
I20250904 07:53:46.988623 6358 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:46.988902 6449 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:46.989284 6449 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 LEADER]: Becoming Leader. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Running, Role: LEADER
I20250904 07:53:46.989539 6449 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:46.991804 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c reported cstate change: term changed from 0 to 1, leader changed from <none> to d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68). New cstate: current_term: 1 leader_uuid: "d67ae74f80964d33aaf83b8e54ed263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:47.085827 6448 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:47.086115 6448 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.086818 6448 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:47.093071 6247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:47.093310 6247 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 0.
I20250904 07:53:47.093663 6172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:47.093719 6284 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:47.093845 6172 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 0.
I20250904 07:53:47.093995 6448 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:53:47.094102 6448 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:47.094192 6448 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:47.096096 6448 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.096536 6448 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 1 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:47.096959 6247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:47.096992 6172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:47.097146 6247 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:47.097182 6172 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:53:47.099704 6247 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 1.
I20250904 07:53:47.099704 6172 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 1.
I20250904 07:53:47.100088 6284 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:47.100318 6448 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:53:47.100672 6448 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:53:47.100950 6448 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.103089 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:47.143929 6172 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:47.143937 6321 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:47.144451 6449 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:47.144672 6452 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:47.151604 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:53:47.161744 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:47.162169 413 tablet_replica.cc:331] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: stopping tablet replica
I20250904 07:53:47.162388 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:47.162585 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.163132 413 tablet_replica.cc:331] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: stopping tablet replica
I20250904 07:53:47.163261 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:47.163394 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.173635 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:53:47.177009 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:47.179183 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.179400 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:47.179829 413 server_base.cc:1047] running on GCE node
W20250904 07:53:47.179836 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:47.180212 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:47.180274 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:47.180301 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972427180298 us; error 0 us; skew 500 ppm
I20250904 07:53:47.180440 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:47.181613 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:47.181815 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:47.181865 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:47.183514 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:47.184664 6474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:47.184995 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.185094 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:47.185214 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:47.203958 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:47.204411 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:47.204926 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:47.205924 6481 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:53:47.210116 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:47.210189 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:53:47.210284 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:53:47.212232 6481 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:47.213697 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:47.213774 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20250904 07:53:47.215461 6481 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.215796 6481 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:47.216071 6481 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:47.216711 6481 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.216943 6481 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.217206 6481 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.217846 6481 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:47.218058 6481 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:47.222752 6481 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.223171 6481 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:47.223412 6481 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:53:47.223927 6481 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.224045 6481 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.224256 6481 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.224658 6481 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250904 07:53:47.229359 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:47.229390 6540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:53:47.236308 6541 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:47.236416 6541 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:47.236685 6541 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:47.237982 6247 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:47.238034 6069 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:47.238479 6455 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:47.240228 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35872
I20250904 07:53:47.247160 6545 mvcc.cc:204] Tried to move back new op lower bound from 7196559061960912896 to 7196559061407424512. Current Snapshot: MvccSnapshot[applied={T|T < 7196559061960912896}]
I20250904 07:53:47.248059 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250904 07:53:47.252285 6516 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:53:47.252923 6455 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:53:47.254261 6541 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:47.255899 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:47.256335 413 tablet_replica.cc:331] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: stopping tablet replica
I20250904 07:53:47.256532 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:47.256711 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.271143 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250904 07:53:47.273995 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:47.276283 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.276363 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.277007 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:47.277410 413 server_base.cc:1047] running on GCE node
I20250904 07:53:47.277583 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:47.277632 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:47.277674 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972427277671 us; error 0 us; skew 500 ppm
I20250904 07:53:47.277797 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:47.278831 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:53:47.279019 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:47.279068 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:47.280556 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.281599 6559 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:47.281872 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.281960 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:47.282053 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:47.294327 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:47.294704 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:47.295148 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:47.298357 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:47.298432 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:47.300105 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:47.300163 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:47.300323 6566 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:53:47.306396 6566 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.306888 6566 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:53:47.307170 6566 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250904 07:53:47.308000 6566 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.308225 6566 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.308511 6566 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.309285 6566 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20250904 07:53:47.312065 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:53:47.312088 6624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:53:47.319528 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250904 07:53:47.324301 6625 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:47.324498 6625 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:47.326309 6625 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:47.329818 6068 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:53:47.331251 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35876
I20250904 07:53:47.331719 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:47.332144 413 tablet_replica.cc:331] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: stopping tablet replica
I20250904 07:53:47.332338 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:47.332556 6625 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:47.332697 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.333500 413 tablet_replica.cc:331] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: stopping tablet replica
I20250904 07:53:47.333612 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:47.333750 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.348147 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250904 07:53:47.351140 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:47.353297 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.353297 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.353982 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:47.354367 413 server_base.cc:1047] running on GCE node
I20250904 07:53:47.354580 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:47.354648 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:47.354712 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972427354693 us; error 0 us; skew 500 ppm
I20250904 07:53:47.354866 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:47.355968 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:53:47.356175 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:47.356246 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:47.357878 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:47.359050 6642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:47.359345 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.359452 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:47.359558 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:47.378572 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:47.379022 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:47.379478 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:47.384995 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:47.385075 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:53:47.386911 6649 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:47.388240 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:47.388320 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:47.393486 6649 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.393900 6649 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:47.394142 6649 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250904 07:53:47.394838 6649 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.395077 6649 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.395344 6649 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:47.396097 6649 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250904 07:53:47.396322 6649 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:47.402531 6649 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.402951 6649 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:47.403251 6649 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250904 07:53:47.404055 6708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:53:47.404052 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:53:47.404052 6649 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.404280 6649 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.404487 6649 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.404982 6649 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20250904 07:53:47.414578 6709 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:47.414741 6709 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:47.415122 6709 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:47.417971 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:47.417976 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250904 07:53:47.421468 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35888
I20250904 07:53:47.422545 6709 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:47.427316 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:47.436036 413 tablet_replica.cc:331] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: stopping tablet replica
I20250904 07:53:47.436225 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:53:47.436479 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:47.450608 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250904 07:53:47.453477 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:47.455722 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.455802 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:47.455977 6720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:47.456226 413 server_base.cc:1047] running on GCE node
I20250904 07:53:47.456566 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:47.456634 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:47.456667 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972427456665 us; error 0 us; skew 500 ppm
I20250904 07:53:47.456797 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:47.457919 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:53:47.458086 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:47.458127 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:47.459645 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.460691 6725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:47.461004 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:47.461087 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:47.461191 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:47.468921 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:47.469270 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:47.469681 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:47.472718 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:47.472779 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:47.474563 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:47.474617 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:47.474722 6732 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:53:47.481703 6732 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:47.482193 6732 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:53:47.482460 6732 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20250904 07:53:47.483489 6732 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.483670 6732 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:53:47.483912 6732 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:47.484505 6732 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20250904 07:53:47.488652 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:53:47.488726 6790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:53:47.495299 6791 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:47.495416 6791 heartbeater.cc:461] Registering TS with master...
W20250904 07:53:47.495608 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:47.495671 6791 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:47.496809 6069 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:47.497931 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35890
W20250904 07:53:47.503083 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.512939 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.518683 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.526106 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.533700 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.546643 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.559226 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.569693 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.581450 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.593837 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.610203 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.629163 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.647771 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.663355 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.683950 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.705395 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.727411 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.749495 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.774286 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.798803 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.826622 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.851066 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.877768 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.906826 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:47.936399 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.966202 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:47.997400 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.029106 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.061885 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.096028 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.129011 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.162887 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.203208 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.239037 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.280025 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.319653 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.359474 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.402334 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.446480 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.492307 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:48.498994 6791 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:48.536077 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.582386 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.630312 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:48.657219 6795 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:48.657387 6795 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:48.658099 6795 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:48.674846 6516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:48.674964 6683 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:48.675457 6726 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:48.675720 6795 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 07:53:48.676064 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:48.726013 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:48.774863 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:48.823603 6800 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:48.823781 6800 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:48.824466 6800 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
W20250904 07:53:48.825704 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:48.830513 6683 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:48.830703 6683 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 1.
I20250904 07:53:48.831033 6478 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:48.831284 6800 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 07:53:48.831382 6800 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:48.831456 6800 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.831518 6804 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:48.831492 6599 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:48.831650 6804 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:48.831882 6599 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 1.
I20250904 07:53:48.832412 6804 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:48.834453 6800 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:48.835114 6800 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:48.835191 6599 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:48.835366 6599 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.835714 6683 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:48.835884 6683 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.838135 6599 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 2.
I20250904 07:53:48.838385 6516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:48.838555 6475 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:48.838644 6516 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 in current term 2: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:48.838565 6683 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 2.
I20250904 07:53:48.838907 6800 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:48.839148 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:48.839351 6800 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:48.839401 6682 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 in current term 2: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:48.839653 6800 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:48.839747 6563 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:48.839962 6804 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 07:53:48.841845 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 1 to 2, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
W20250904 07:53:48.876773 6745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33860: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:48.917551 6800 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:53:48.917762 6800 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:48.918433 6800 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:48.918999 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:48.920574 6814 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:53:48.920732 6814 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:48.921525 6814 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:48.928174 6765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:48.928409 6765 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 1.
W20250904 07:53:48.929426 6494 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59916: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:48.929570 6478 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243
I20250904 07:53:48.929965 6800 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 07:53:48.929991 6516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:48.930119 6800 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:53:48.930238 6800 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.932148 6800 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:48.932464 6516 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 2: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:48.932802 6800 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:48.933087 6765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:48.933281 6765 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 1.
I20250904 07:53:48.933292 6764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:53:48.933328 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:48.933470 6764 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.933521 6682 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 1 FOLLOWER]: Advancing to term 2
I20250904 07:53:48.933712 6646 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:48.935340 6764 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 2.
I20250904 07:53:48.935340 6682 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 2.
I20250904 07:53:48.935524 6814 raft_consensus.cc:2762] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250904 07:53:48.935689 6478 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:48.935935 6800 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 07:53:48.936174 6800 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:48.936417 6800 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:48.941399 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 1 to 2, leader changed from d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250904 07:53:48.983234 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:49.043403 6682 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250904 07:53:49.043447 6764 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250904 07:53:49.043990 6807 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250904 07:53:49.044188 6800 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250904 07:53:49.050068 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:53:49.059756 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:49.059984 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:49.060151 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 LEADER]: Raft consensus shutting down.
I20250904 07:53:49.060429 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:49.061159 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:49.061309 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 LEADER]: Raft consensus shutting down.
I20250904 07:53:49.061517 413 pending_rounds.cc:70] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Trying to abort 1 pending ops.
I20250904 07:53:49.061698 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7196559068529410048 op_type: NO_OP noop_request { }
I20250904 07:53:49.061829 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:53:49.061955 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:49.066478 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:53:49.069481 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:49.071564 6828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:49.071648 6829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:49.072046 6831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:49.072651 413 server_base.cc:1047] running on GCE node
I20250904 07:53:49.072906 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:49.072983 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:49.073047 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972429073028 us; error 0 us; skew 500 ppm
I20250904 07:53:49.073200 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:49.074265 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:49.074463 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:49.074534 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:49.076037 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250904 07:53:49.077193 6836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:49.077522 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:49.077634 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:49.077747 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:49.094660 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:49.095288 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:49.095925 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:49.098263 6843 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:53:49.102550 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:49.102631 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20250904 07:53:49.102710 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:53:49.104624 6843 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:49.106105 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:49.106180 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20250904 07:53:49.110680 6843 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250904 07:53:49.111203 6843 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:49.111531 6843 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250904 07:53:49.112269 6843 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:49.112582 6843 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:49.112944 6843 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:49.113646 6843 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250904 07:53:49.113974 6843 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:49.120378 6843 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:49.120765 6843 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:49.121048 6843 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20250904 07:53:49.121757 6843 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:49.121945 6843 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:49.122217 6843 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:49.122732 6843 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:49.123684 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:49.123708 6902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:53:49.131884 6903 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:49.132041 6903 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:49.132352 6903 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:49.133987 6068 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:49.135222 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48910
W20250904 07:53:49.136119 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.142438 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.147955 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.156500 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.161746 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.172181 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.182667 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.193042 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.205457 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.218056 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.233527 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.248023 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.266628 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.282968 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.300498 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.320428 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.339206 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.359427 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.383824 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.404567 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.428635 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.453965 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.477789 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.503474 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.530457 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.557200 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.584975 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.618423 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.652256 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.684072 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.717914 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.750623 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.786459 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.822522 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.861450 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.902104 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.941264 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:49.979933 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.020658 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.062460 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.107131 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.136406 6903 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:50.152877 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.200057 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.247802 6579 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34024: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.295614 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.342901 6910 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:50.343065 6910 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:50.343674 6910 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:50.344189 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:50.344374 6682 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 2.
W20250904 07:53:50.344619 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.344765 6563 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:50.345074 6910 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 07:53:50.345168 6910 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:50.345248 6910 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.347103 6910 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:50.347556 6910 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 3 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:50.347940 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:50.348076 6682 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.349375 6877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:50.349467 6876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:50.349637 6877 raft_consensus.cc:2408] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250904 07:53:50.350323 6682 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 3.
I20250904 07:53:50.350621 6563 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:50.350838 6910 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 07:53:50.351151 6910 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 LEADER]: Becoming Leader. State: Replica: d888acb801154571b952c5dba8b792d7, State: Running, Role: LEADER
I20250904 07:53:50.351378 6910 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:50.353451 6068 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 reported cstate change: term changed from 2 to 3, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to d888acb801154571b952c5dba8b792d7 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "d888acb801154571b952c5dba8b792d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:50.393914 6877 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.393944 6682 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Refusing update from remote peer d888acb801154571b952c5dba8b792d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 07:53:50.394423 6913 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250904 07:53:50.396183 6877 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Refusing update from remote peer d888acb801154571b952c5dba8b792d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250904 07:53:50.396615 6913 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250904 07:53:50.396770 6913 consensus_queue.cc:1230] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Peer 6a2e703f66a34a598ab3d9646fa5db40 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250904 07:53:50.397526 6877 pending_rounds.cc:85] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Aborting all ops after (but not including) 3
I20250904 07:53:50.397608 6877 pending_rounds.cc:107] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20250904 07:53:50.397771 6877 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250904 07:53:50.399544 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:53:50.412648 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:50.412882 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:50.413080 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 LEADER]: Raft consensus shutting down.
I20250904 07:53:50.413301 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:50.417799 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:53:50.420595 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:50.422773 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:50.423465 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:50.428422 413 server_base.cc:1047] running on GCE node
W20250904 07:53:50.428988 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:50.429306 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:50.429360 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:50.429383 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972430429381 us; error 0 us; skew 500 ppm
I20250904 07:53:50.429493 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:50.430536 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:53:50.430694 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:50.430734 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:50.432240 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:50.433322 6938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:50.433604 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:50.433684 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:50.433768 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:50.446076 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:50.446482 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:50.446880 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:50.447721 6945 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:53:50.449972 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:50.450042 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:50.450124 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:53:50.451802 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:50.451866 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:50.451946 6945 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:53:50.459806 6945 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:50.460189 6945 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:53:50.460513 6945 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20250904 07:53:50.461319 6945 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:50.461529 6945 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:53:50.461799 6945 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:50.462574 6945 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:50.466580 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:53:50.466603 7003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:53:50.474767 7004 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:50.474884 7004 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:50.475170 7004 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:50.476147 6068 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:53:50.477334 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48926
W20250904 07:53:50.478274 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.480573 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.483918 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.491299 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.499492 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.508745 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.519425 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.530831 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.540228 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.546134 6924 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:50.546269 6924 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.546923 6924 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:50.551656 6920 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:50.551844 6920 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.552616 6920 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:50.553020 7014 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:50.553169 7014 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.553216 6764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:50.553445 6764 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 2.
I20250904 07:53:50.553253 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:50.553838 7014 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:50.553908 6682 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 2.
W20250904 07:53:50.554309 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.554459 6840 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:50.554730 6924 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 07:53:50.554834 6924 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:50.554944 6924 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.555128 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:50.555303 6682 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 2.
I20250904 07:53:50.555737 6646 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:50.555987 6920 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 07:53:50.556030 6729 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:50.556103 6920 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:50.556216 6920 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.556347 7014 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 07:53:50.556434 7014 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:50.556519 7014 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 2 FOLLOWER]: Advancing to term 3
I20250904 07:53:50.557886 6924 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.558782 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:50.559358 6924 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 3 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:50.559301 7014 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.559644 6920 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:50.560132 6682 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 3: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:50.560412 6920 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 3 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:50.561218 6765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:50.561301 6763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:53:50.561376 6764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:53:50.561496 6765 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 3: Already voted for candidate d67ae74f80964d33aaf83b8e54ed263c in this term.
I20250904 07:53:50.561868 6840 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40; no voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:53:50.562180 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:50.562369 6682 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 3: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:50.562752 7014 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 3 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:50.562775 6924 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250904 07:53:50.568085 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:50.572942 6877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:50.573045 6876 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:50.573184 6877 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 3: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:50.573128 6875 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:50.573256 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:50.573338 6875 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 3: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:50.573524 6643 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:53:50.573827 6920 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250904 07:53:50.573863 6726 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:50.574074 7014 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250904 07:53:50.581656 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.597294 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.615638 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.631166 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.652050 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.673643 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.695144 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.718950 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.739571 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.764349 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.790381 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.813898 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.839725 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.870906 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.899487 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.928288 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.960310 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:50.991931 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.026803 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.062099 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.095757 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.131646 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.169895 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.206686 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.243491 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.285858 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.327592 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.371631 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.413453 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.456116 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:51.478650 7004 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:51.498701 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.544759 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.592643 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.642580 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.694036 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.744935 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.796972 6662 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42438: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:51.851177 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:51.899144 7018 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d888acb801154571b952c5dba8b792d7)
I20250904 07:53:51.899365 7018 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:51.900046 7018 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:51.901656 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
W20250904 07:53:51.903693 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:51.906268 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:51.906481 6978 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 3.
I20250904 07:53:51.906683 7021 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d888acb801154571b952c5dba8b792d7)
I20250904 07:53:51.906845 7021 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:51.906911 6643 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:51.907188 7018 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 07:53:51.907308 7018 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d888acb801154571b952c5dba8b792d7)
I20250904 07:53:51.907438 7018 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:51.907591 7021 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:51.908231 6682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:51.910095 7018 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:51.910442 6682 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 4: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:51.910689 7018 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 4 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:51.911096 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:51.911144 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:51.911304 6874 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:51.911325 6978 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:51.913676 6874 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 4.
I20250904 07:53:51.914177 6643 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:51.914223 6978 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 4.
I20250904 07:53:51.914495 7018 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Leader election won for term 4
I20250904 07:53:51.914922 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:51.914930 7018 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:53:51.915239 6978 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 4: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:51.915362 7018 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:51.915614 6837 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40; no voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7
I20250904 07:53:51.915848 7021 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250904 07:53:51.917737 6068 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 3 to 4, leader changed from d888acb801154571b952c5dba8b792d7 (127.0.103.66) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 4 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:51.957429 6874 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250904 07:53:51.957429 6978 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250904 07:53:51.958079 7018 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 07:53:51.958305 7023 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 07:53:51.965449 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:53:51.977743 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:51.978013 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:51.978174 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:51.978413 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:51.978936 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:51.979108 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 LEADER]: Raft consensus shutting down.
I20250904 07:53:51.979367 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:51.985921 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:53:51.989123 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:51.991621 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:51.991641 7039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:51.992064 7041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:51.992528 413 server_base.cc:1047] running on GCE node
I20250904 07:53:51.992730 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:51.992794 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:51.992851 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972431992834 us; error 0 us; skew 500 ppm
I20250904 07:53:51.993041 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:51.994148 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:53:51.994350 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:51.994415 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:51.996009 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:51.997246 7046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:51.997584 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:51.997710 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:51.997848 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:52.019655 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:52.020195 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:52.020714 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:52.021772 7053 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:53:52.026069 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:52.026165 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:53:52.026274 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:53:52.028164 7053 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:52.029827 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:52.029914 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:52.037056 7053 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:52.037561 7053 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:52.037837 7053 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250904 07:53:52.038502 7053 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:52.038673 7053 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:52.038928 7053 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:52.039721 7053 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:52.040074 7053 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:52.045977 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:53:52.046036 7112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:53:52.050457 7053 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:52.050886 7053 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:52.051692 7053 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250904 07:53:52.052627 7053 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:52.052821 7053 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:52.053102 7053 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:52.053707 7053 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:52.060770 7113 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:52.060968 7113 heartbeater.cc:461] Registering TS with master...
W20250904 07:53:52.061139 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:52.061383 7113 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:52.062587 6068 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:52.063956 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48942
I20250904 07:53:52.065111 7113 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:52.066018 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.072451 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.080118 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.087534 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.095824 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.105966 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.117617 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.131209 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.147245 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.163017 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:52.175256 7119 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:52.175436 7119 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:52.176120 7119 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:52.176431 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:52.176652 6874 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 3.
W20250904 07:53:52.176813 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:52.177143 6726 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:53:52.177449 7119 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 07:53:52.177594 7119 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:52.177698 7119 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:52.180727 7119 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:52.181367 7119 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 4 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:52.181677 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:52.181874 6874 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:52.183166 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:52.183293 7085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:52.183360 7086 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 3.
I20250904 07:53:52.184347 6874 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 4.
I20250904 07:53:52.184671 6726 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243
I20250904 07:53:52.184957 7119 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Leader election won for term 4
I20250904 07:53:52.185335 7119 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 LEADER]: Becoming Leader. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Running, Role: LEADER
I20250904 07:53:52.185573 7119 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:52.187701 6068 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c reported cstate change: term changed from 2 to 4, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68). New cstate: current_term: 4 leader_uuid: "d67ae74f80964d33aaf83b8e54ed263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: HEALTHY } } }
W20250904 07:53:52.195829 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.214633 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.231381 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.251613 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.271538 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.291289 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.313228 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.335924 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.357638 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.383842 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.409796 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.438655 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.468142 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.499115 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.529826 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.559939 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.590742 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:52.599718 7085 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 3 FOLLOWER]: Advancing to term 4
I20250904 07:53:52.602011 7085 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250904 07:53:52.602591 7119 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 07:53:52.606720 6874 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250904 07:53:52.607445 7121 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250904 07:53:52.622359 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.659659 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.692461 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.729307 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.766742 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.804766 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.843818 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.885221 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.928189 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:52.970958 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.014240 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.055974 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.100915 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.146998 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.191978 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.238837 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.287243 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.337303 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.386224 6856 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59776: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.438519 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:53.463178 7133 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:53.463362 7133 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.464048 7133 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:53.464572 7134 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:53.464846 7134 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.464959 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:53.465575 6978 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 4.
I20250904 07:53:53.466037 6837 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:53.466341 7133 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250904 07:53:53.466452 7133 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:53.466547 7133 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:53.469514 7133 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.470130 7134 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:53.472971 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:53.473085 7133 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 5 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:53.473187 6978 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:53.473973 7085 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:53.474167 7085 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 4.
I20250904 07:53:53.474912 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:53.475102 7086 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:53.476096 6978 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 5.
I20250904 07:53:53.476459 6837 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:53.476728 7133 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader election won for term 5
I20250904 07:53:53.477991 7086 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 5.
I20250904 07:53:53.478891 7133 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:53.479194 7133 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.479807 6874 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:53.480099 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:53.480340 7086 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 in current term 5: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:53:53.480751 6942 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:53.481096 7134 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250904 07:53:53.481587 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 4 to 5, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 5 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
W20250904 07:53:53.493242 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:53.547786 6978 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 9. (index mismatch)
I20250904 07:53:53.547788 7086 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 9. (index mismatch)
I20250904 07:53:53.548389 7133 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250904 07:53:53.548597 7141 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250904 07:53:53.559291 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:53:53.567551 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:53.567800 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:53.567970 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 LEADER]: Raft consensus shutting down.
I20250904 07:53:53.568259 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:53.581982 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:53:53.585047 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:53.587355 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:53.587885 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:53.589516 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:53.589592 413 server_base.cc:1047] running on GCE node
I20250904 07:53:53.589857 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:53.589924 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:53.589980 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972433589963 us; error 0 us; skew 500 ppm
I20250904 07:53:53.590137 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:53.591305 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:53:53.591502 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:53.591569 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:53.593278 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:53.594376 7165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:53.594695 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:53.594808 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:53.594921 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:53.614403 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:53.614881 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:53.615339 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:53.616362 7172 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:53:53.618886 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:53.618976 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:53.619102 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:53:53.620818 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:53.620927 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:53.621026 7172 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:53:53.628625 7172 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:53.629076 7172 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:53:53.629350 7172 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20250904 07:53:53.630105 7172 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:53.630281 7172 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:53:53.630602 7172 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:53.631325 7172 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:53:53.635689 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:53:53.635725 7230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:53:53.643568 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:53:53.645215 7231 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:53.645530 7231 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:53.646991 7231 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:53.648465 6069 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:53.649963 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48946
I20250904 07:53:53.653381 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:53.653573 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:53.653719 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:53.653868 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:53.654433 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:53.654546 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 LEADER]: Raft consensus shutting down.
I20250904 07:53:53.654775 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:53.659358 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:53:53.662518 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:53.664800 7236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:53.664929 7235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:53.665022 7238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:53.665544 413 server_base.cc:1047] running on GCE node
I20250904 07:53:53.665758 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:53.665841 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:53.665884 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972433665864 us; error 0 us; skew 500 ppm
I20250904 07:53:53.666030 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:53.667191 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:53.667407 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:53.667474 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:53.669281 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:53.670560 7243 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:53.670904 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:53.671003 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:53.671134 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:53.682958 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:53.683488 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:53.683977 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:53.689321 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:53.689404 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:53:53.691397 7250 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:53.693135 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:53.693217 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.002s
I20250904 07:53:53.706336 7250 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=1 applied=10 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:53.706851 7250 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:53.707119 7250 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.001s
I20250904 07:53:53.707892 7250 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.708107 7250 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:53.708360 7250 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:53.709003 7250 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20250904 07:53:53.709247 7250 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:53.713116 7309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:53:53.713116 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:53.720669 7250 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:53.721166 7250 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:53.721470 7250 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20250904 07:53:53.722502 7250 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:53.722676 7250 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:53.722959 7250 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:53.723508 7250 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:53:53.723670 7310 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:53.723820 7310 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:53.724130 7310 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:53.725154 6069 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
W20250904 07:53:53.725298 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:53.726416 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48960
I20250904 07:53:53.727321 7310 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:53.730748 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.736111 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.745894 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.755357 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.763969 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.776610 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.788988 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.798450 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.814965 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.827554 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.842208 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.860260 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.878927 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.895509 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.913666 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.932370 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.955106 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:53.976085 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.000769 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.026690 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.050964 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.078850 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.104638 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.132987 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.162840 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.194844 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.229578 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.259570 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.292557 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.327158 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.363111 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.396944 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.437451 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.477313 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.515189 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:54.542734 7315 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:53:54.542917 7315 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:54.543706 7315 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
W20250904 07:53:54.557752 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:54.561677 7283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:54.561932 7283 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 4.
I20250904 07:53:54.562031 7205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:54.562211 7205 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 4.
I20250904 07:53:54.562341 7047 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:54.562579 7315 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250904 07:53:54.562660 7315 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:53:54.562723 7315 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:54.564934 7315 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:54.565377 7315 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 5 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:54.565793 7205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:53:54.565806 7283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:54.565963 7205 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:54.565984 7283 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 4 FOLLOWER]: Advancing to term 5
I20250904 07:53:54.567883 7205 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 5.
I20250904 07:53:54.567891 7283 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 5.
I20250904 07:53:54.568212 7047 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:54.568470 7315 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader election won for term 5
I20250904 07:53:54.568809 7315 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:53:54.569104 7315 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:54.570983 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 4 to 5, leader changed from d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 5 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250904 07:53:54.600473 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.640381 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:54.653016 7231 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:54.684612 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.729691 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.775740 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.824141 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.873191 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.919251 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:54.967569 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.018522 6958 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34038: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.070552 7066 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50186: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:55.078739 7205 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250904 07:53:55.079311 7326 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250904 07:53:55.083096 7283 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250904 07:53:55.083649 7326 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250904 07:53:55.101027 7333 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:55.101226 7333 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.102011 7333 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
W20250904 07:53:55.122885 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:55.124706 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:55.124927 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:55.125366 7247 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40; no voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7
I20250904 07:53:55.125612 7333 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250904 07:53:55.142657 7326 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:55.142722 7340 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:55.142906 7340 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.142932 7326 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.143549 7340 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:55.143679 7326 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:55.144150 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:55.144387 7086 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 5.
I20250904 07:53:55.144964 6942 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:55.145272 7340 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250904 07:53:55.145383 7340 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:55.145519 7340 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 5 FOLLOWER]: Advancing to term 6
I20250904 07:53:55.146595 7283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:55.146811 7283 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 5.
I20250904 07:53:55.147327 7047 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:55.147655 7326 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250904 07:53:55.147781 7326 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:55.147874 7326 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Advancing to term 6
I20250904 07:53:55.148780 7340 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.150130 7086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:55.150393 7340 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 6 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:55.151013 7326 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.151455 7086 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 in current term 6: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:55.152478 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:55.152733 6978 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 6: Already voted for candidate d888acb801154571b952c5dba8b792d7 in this term.
I20250904 07:53:55.152769 7283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:55.153205 7281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:55.153286 7283 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 5.
I20250904 07:53:55.152576 7326 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 6 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:55.152961 7282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:55.152915 6977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:55.153810 7282 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Advancing to term 6
I20250904 07:53:55.153874 6977 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 6: Already voted for candidate d888acb801154571b952c5dba8b792d7 in this term.
I20250904 07:53:55.154233 7047 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7
I20250904 07:53:55.154462 7326 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250904 07:53:55.156011 7282 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 6.
I20250904 07:53:55.156373 6939 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243
I20250904 07:53:55.156649 7340 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Leader election won for term 6
I20250904 07:53:55.157043 7340 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 LEADER]: Becoming Leader. State: Replica: d888acb801154571b952c5dba8b792d7, State: Running, Role: LEADER
I20250904 07:53:55.157338 7340 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.159473 6068 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 reported cstate change: term changed from 5 to 6, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to d888acb801154571b952c5dba8b792d7 (127.0.103.66). New cstate: current_term: 6 leader_uuid: "d888acb801154571b952c5dba8b792d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:55.175679 7282 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Refusing update from remote peer d888acb801154571b952c5dba8b792d7: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20250904 07:53:55.175680 7086 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Refusing update from remote peer d888acb801154571b952c5dba8b792d7: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
I20250904 07:53:55.176221 7344 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250904 07:53:55.176434 7340 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250904 07:53:55.182148 7330 mvcc.cc:204] Tried to move back new op lower bound from 7196559094474919936 to 7196559094406467584. Current Snapshot: MvccSnapshot[applied={T|T < 7196559094474919936}]
I20250904 07:53:55.186218 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:53:55.193758 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:55.193982 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:55.194155 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 LEADER]: Raft consensus shutting down.
I20250904 07:53:55.194463 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:55.198911 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:53:55.201929 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:55.204118 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:55.204188 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:55.204695 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:55.205083 413 server_base.cc:1047] running on GCE node
I20250904 07:53:55.205348 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:55.205416 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:55.205475 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972435205457 us; error 0 us; skew 500 ppm
I20250904 07:53:55.205636 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:55.206796 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:53:55.207002 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:55.207067 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:55.208734 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:55.210000 7364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:55.210330 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:55.210444 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:55.210561 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:55.225059 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:55.225538 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:55.226105 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:55.227270 7371 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:53:55.229961 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:55.230077 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:55.230203 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:53:55.232010 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:55.232097 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:55.232167 7371 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:53:55.243832 7371 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:55.244339 7371 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:53:55.244603 7371 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250904 07:53:55.245370 7371 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.245554 7371 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:53:55.245855 7371 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 6.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.246554 7371 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:55.248448 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:53:55.248492 7429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
W20250904 07:53:55.250717 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:55.255405 7430 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:55.255549 7430 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:55.255898 7430 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:55.259255 6069 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:53:55.260619 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48970
I20250904 07:53:55.261289 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:53:55.270148 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:55.270381 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:55.270552 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 LEADER]: Raft consensus shutting down.
I20250904 07:53:55.270839 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:55.271492 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:55.271632 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:55.271842 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:55.276154 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:53:55.279237 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:55.281472 7435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:55.282182 7438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:55.282294 413 server_base.cc:1047] running on GCE node
W20250904 07:53:55.282182 7436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:55.282737 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:55.282820 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:55.282886 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972435282867 us; error 0 us; skew 500 ppm
I20250904 07:53:55.283056 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:55.284194 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:53:55.284390 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:55.284461 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:55.286154 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:55.287338 7443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:55.287638 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:55.287761 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:55.287878 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:55.296986 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:55.297521 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:55.297978 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:55.303110 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:55.303201 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:53:55.305205 7450 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:55.306807 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:55.306887 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250904 07:53:55.317481 7450 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:55.317912 7450 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:55.318172 7450 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20250904 07:53:55.318856 7450 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.319047 7450 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:55.319326 7450 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 6.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:55.320133 7450 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250904 07:53:55.320415 7450 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:55.323336 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:53:55.323357 7508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:53:55.332204 7450 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:55.332764 7450 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:55.333112 7450 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250904 07:53:55.333798 7450 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:55.333969 7450 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:55.334215 7450 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:55.334704 7450 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250904 07:53:55.338898 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:55.340183 7510 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:55.340313 7510 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:55.340643 7510 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:55.341750 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
W20250904 07:53:55.342613 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:55.343288 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48980
I20250904 07:53:55.344211 7510 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:55.351300 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.356784 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.365301 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.371666 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.380275 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.392649 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.402177 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.418092 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.431533 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.448177 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.467114 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.485669 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.505390 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.527570 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.549269 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.571108 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.593184 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.614846 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.639803 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.668315 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.695407 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.723369 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.754562 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.782404 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.811297 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.841856 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.872767 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.904677 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.940169 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:55.975051 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.010943 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.051330 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.089035 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.129011 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.169277 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.210143 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.253932 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:56.262166 7430 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:56.298116 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.342888 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.386729 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.435081 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.481984 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.529887 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.580529 7384 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34090: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.631716 7263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59808: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:56.681871 7516 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d888acb801154571b952c5dba8b792d7)
I20250904 07:53:56.682111 7516 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:56.682797 7516 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
W20250904 07:53:56.682833 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:56.698236 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:56.698239 7404 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:56.698523 7483 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 6.
I20250904 07:53:56.698529 7404 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 6.
I20250904 07:53:56.698969 7247 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:56.699236 7516 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250904 07:53:56.699342 7516 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d888acb801154571b952c5dba8b792d7)
I20250904 07:53:56.699433 7516 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Advancing to term 7
I20250904 07:53:56.701501 7516 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:56.702030 7516 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:56.702486 7404 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:56.702666 7404 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 6 FOLLOWER]: Advancing to term 7
I20250904 07:53:56.704559 7404 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 7.
I20250904 07:53:56.704882 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:56.705065 7483 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Advancing to term 7
I20250904 07:53:56.705032 7244 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:53:56.705331 7516 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Leader election won for term 7
I20250904 07:53:56.705845 7516 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:56.706130 7516 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 6.12, Last appended by leader: 12, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:56.707854 7483 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 7.
I20250904 07:53:56.708767 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 6 to 7, leader changed from d888acb801154571b952c5dba8b792d7 (127.0.103.66) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 7 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:56.744431 7404 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 6 index: 12. Preceding OpId from leader: term: 7 index: 14. (index mismatch)
I20250904 07:53:56.744431 7483 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 6 index: 12. Preceding OpId from leader: term: 7 index: 14. (index mismatch)
I20250904 07:53:56.745115 7521 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250904 07:53:56.745402 7516 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250904 07:53:56.752419 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:53:56.760772 7516 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:56.761046 7516 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.761278 7538 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:56.761469 7538 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.761874 7516 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:56.762254 7538 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:56.769438 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:56.769709 7483 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 5.
I20250904 07:53:56.770134 7247 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:56.770380 7516 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250904 07:53:56.770454 7516 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:56.770514 7516 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 5 FOLLOWER]: Advancing to term 6
I20250904 07:53:56.773406 7516 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.773778 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:56.774178 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:56.774335 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:56.774515 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:56.774633 7516 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:56.774857 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:56.775004 7483 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 5 FOLLOWER]: Advancing to term 6
I20250904 07:53:56.775247 7482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
W20250904 07:53:56.775837 7247 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 32 similar messages]
W20250904 07:53:56.777328 7247 leader_election.cc:336] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250904 07:53:56.777638 7247 leader_election.cc:336] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250904 07:53:56.777997 7483 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 6.
I20250904 07:53:56.778345 7247 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:53:56.778549 7516 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Leader election won for term 6
I20250904 07:53:56.778704 7516 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:56.778998 7516 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.780827 7282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:56.781545 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 5 to 6, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 6 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250904 07:53:56.785248 7247 connection.cc:537] server connection from 127.0.0.1:59844 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250904 07:53:56.785411 7544 leader_election.cc:336] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935): Service unavailable: reactor is shutting down (error 108)
I20250904 07:53:56.785575 7544 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
W20250904 07:53:56.785843 7544 raft_consensus.cc:2684] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250904 07:53:56.797765 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:53:56.800645 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:56.802996 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:56.803084 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:56.803390 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:56.803773 413 server_base.cc:1047] running on GCE node
I20250904 07:53:56.804000 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:56.804060 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:56.804102 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972436804088 us; error 0 us; skew 500 ppm
I20250904 07:53:56.804214 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:56.805377 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:53:56.805557 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:56.805613 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:56.807161 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:56.808281 7554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:56.808586 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:56.808702 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:56.808804 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:56.832336 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:56.832790 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:56.833320 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:56.834358 7561 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:53:56.836701 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:56.836796 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:56.836946 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:53:56.838759 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:56.838843 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:56.838915 7561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:53:56.847252 7561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:56.847652 7561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:53:56.847960 7561 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250904 07:53:56.848671 7561 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.848850 7561 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:53:56.849172 7561 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.849915 7561 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20250904 07:53:56.853531 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:53:56.853571 7619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:53:56.860471 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:53:56.862709 7620 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:56.862897 7620 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:56.863268 7620 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:56.864612 6069 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:56.866000 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48988
I20250904 07:53:56.871326 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:56.871546 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:56.871709 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 LEADER]: Raft consensus shutting down.
I20250904 07:53:56.871937 413 pending_rounds.cc:70] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Trying to abort 1 pending ops.
I20250904 07:53:56.872011 413 pending_rounds.cc:77] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Aborting op as it isn't in flight: id { term: 6 index: 9 } timestamp: 7196559101049110528 op_type: NO_OP noop_request { }
I20250904 07:53:56.872131 413 raft_consensus.cc:2887] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:53:56.872241 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:56.872948 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:56.873066 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 LEADER]: Raft consensus shutting down.
I20250904 07:53:56.873274 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:56.887622 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:53:56.891002 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:56.893323 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:56.893323 7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:56.894022 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:56.894117 413 server_base.cc:1047] running on GCE node
I20250904 07:53:56.894413 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:56.894495 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:56.894558 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972436894538 us; error 0 us; skew 500 ppm
I20250904 07:53:56.894713 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:56.895841 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:56.896049 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:56.896122 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:56.897975 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:53:56.899307 7632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:56.899586 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:56.899688 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:56.899807 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:56.920313 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:56.920769 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:56.921304 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:56.926502 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:56.926586 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:53:56.928504 7639 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:56.930122 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:56.930214 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250904 07:53:56.942561 7639 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=1 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:56.942970 7639 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:56.943282 7639 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250904 07:53:56.943982 7639 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:56.944155 7639 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:56.944416 7639 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 7.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:56.945101 7639 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250904 07:53:56.945359 7639 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:56.946557 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:56.946569 7697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
W20250904 07:53:56.948954 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:56.956329 7698 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:56.956506 7698 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:56.956960 7698 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:56.958298 6069 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:56.960016 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49000
I20250904 07:53:56.960419 7639 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=8 ignored=0} inserts{seen=4 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250904 07:53:56.960988 7639 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:56.961265 7639 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20250904 07:53:56.962023 7639 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.962252 7639 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:56.962481 7639 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:56.962790 7698 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:56.962878 7639 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:53:56.970721 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:56.976318 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.984722 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:56.991048 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.002746 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.011301 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.020754 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.036787 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.052505 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.068228 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.083055 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.099593 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.119251 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.141505 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.159206 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.178812 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.202494 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.223172 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.245054 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.270607 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.297377 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.326179 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.353425 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.383380 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.414309 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.447861 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.479770 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.513490 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.547878 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.580747 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.614447 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.651015 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.687023 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.726780 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:57.766872 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.808640 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.848572 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:57.867564 7620 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:53:57.891713 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.935416 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:57.982198 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.028573 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.073612 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.122421 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.170795 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.222608 7463 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50248: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.275674 7574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51376: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:58.295909 7705 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:58.296092 7705 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.296844 7705 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:58.302981 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:58.303192 7483 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 6.
I20250904 07:53:58.303614 7636 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:58.303889 7705 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250904 07:53:58.304045 7705 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:58.304152 7705 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 6 FOLLOWER]: Advancing to term 7
I20250904 07:53:58.306120 7594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:53:58.306315 7594 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 5.
I20250904 07:53:58.306465 7705 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.306998 7705 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:58.307452 7594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:53:58.307416 7483 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:53:58.307637 7594 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 5 FOLLOWER]: Advancing to term 7
I20250904 07:53:58.307636 7483 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 6 FOLLOWER]: Advancing to term 7
I20250904 07:53:58.310678 7483 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 7.
I20250904 07:53:58.310678 7594 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 7.
I20250904 07:53:58.311061 7636 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:53:58.311355 7705 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Leader election won for term 7
I20250904 07:53:58.311837 7705 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:53:58.312124 7705 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.314466 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 6 to 7. New cstate: current_term: 7 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250904 07:53:58.331604 7594 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 11. (index mismatch)
I20250904 07:53:58.331636 7483 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 11. (index mismatch)
I20250904 07:53:58.332156 7705 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250904 07:53:58.332412 7710 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250904 07:53:58.339370 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:53:58.348088 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:58.348301 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.348462 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:58.348632 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.360535 7725 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:58.360697 7725 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.361586 7725 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:58.361856 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
W20250904 07:53:58.364153 7444 leader_election.cc:336] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779): Network error: Client connection negotiation failed: client connection to 127.0.103.66:32779: connect: Connection refused (error 111)
I20250904 07:53:58.365772 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:58.368238 7733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.368418 7734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:58.370203 7672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 8 candidate_status { last_received { term: 7 index: 15 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:58.370493 7672 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 7.
I20250904 07:53:58.370951 7444 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:53:58.371289 7725 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250904 07:53:58.371368 7725 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:53:58.371467 7725 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Advancing to term 8
I20250904 07:53:58.373430 7725 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.374053 7725 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:58.375007 7672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 8 candidate_status { last_received { term: 7 index: 15 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:58.375201 7672 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Advancing to term 8
W20250904 07:53:58.375257 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.375530 7444 leader_election.cc:336] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779): Network error: Client connection negotiation failed: client connection to 127.0.103.66:32779: connect: Connection refused (error 111)
I20250904 07:53:58.377192 413 server_base.cc:1047] running on GCE node
I20250904 07:53:58.377416 7672 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 8.
I20250904 07:53:58.377470 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:58.377539 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:58.377604 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972438377582 us; error 0 us; skew 500 ppm
I20250904 07:53:58.377790 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:58.377790 7444 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:53:58.378057 7725 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Leader election won for term 8
I20250904 07:53:58.378217 7725 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:53:58.378480 7725 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 7.15, Last appended by leader: 15, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.380676 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 7 to 8, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 8 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
I20250904 07:53:58.385556 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:53:58.385766 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:58.385835 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:58.387475 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:58.394891 7742 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:58.395196 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:58.395295 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:58.395397 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:58.402314 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:58.402747 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:58.403191 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:58.404235 7749 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:53:58.406466 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:58.406533 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:58.406623 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:53:58.408281 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:58.408336 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:53:58.408406 7749 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:53:58.421828 7749 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:58.422206 7749 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:53:58.422452 7749 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250904 07:53:58.423107 7749 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.423264 7749 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:53:58.423557 7749 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 7.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.423821 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:53:58.423902 7806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:53:58.424356 7749 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20250904 07:53:58.426142 7652 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59858: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:58.435170 7808 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:58.435303 7808 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:58.435699 7808 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
W20250904 07:53:58.436576 7762 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52322: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:53:58.436748 6069 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:53:58.437849 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58662
I20250904 07:53:58.444638 7672 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 7 index: 15. Preceding OpId from leader: term: 8 index: 17. (index mismatch)
I20250904 07:53:58.445206 7718 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250904 07:53:58.449046 7782 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 7 FOLLOWER]: Advancing to term 8
I20250904 07:53:58.450002 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:53:58.452587 7782 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 8 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 7 index: 15. Preceding OpId from leader: term: 8 index: 17. (index mismatch)
I20250904 07:53:58.453461 7718 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250904 07:53:58.456768 7808 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:58.459482 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:58.459687 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.459919 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:58.460110 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.460891 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.461045 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 LEADER]: Raft consensus shutting down.
I20250904 07:53:58.461355 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.465757 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:53:58.468678 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:58.471061 7818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.471061 7819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.471446 7821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:58.471680 413 server_base.cc:1047] running on GCE node
I20250904 07:53:58.471946 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:58.472024 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:58.472083 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972438472066 us; error 0 us; skew 500 ppm
I20250904 07:53:58.472229 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:58.473434 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:53:58.473634 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:58.473701 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:58.475265 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250904 07:53:58.476419 7826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:58.476773 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:53:58.476943 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:58.477082 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:58.486472 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:58.486943 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:58.487511 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:58.492813 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:58.492914 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:53:58.494814 7833 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:58.496320 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:58.496405 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:58.510402 7833 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:58.510879 7833 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:58.511190 7833 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250904 07:53:58.511780 7833 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.511945 7833 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:53:58.512190 7833 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 8.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.512986 7833 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20250904 07:53:58.513248 7833 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:53:58.515476 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:53:58.515532 7891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:53:58.522972 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:53:58.524405 7892 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:58.524641 7892 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:58.525310 7892 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:58.526639 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:58.529230 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58668
I20250904 07:53:58.532200 7833 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=5 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:58.532845 7833 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:53:58.533192 7833 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20250904 07:53:58.533895 7833 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.534161 7833 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
W20250904 07:53:58.534530 7636 consensus_peers.cc:489] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 -> Peer 47e41d546b88492da3428adbff442243 (127.0.103.67:44843): Couldn't send request to peer 47e41d546b88492da3428adbff442243. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:53:58.534538 7833 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.535037 7892 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:58.535116 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:58.535348 7833 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:53:58.535380 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.535557 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:58.535787 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.540313 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:53:58.543323 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:58.545519 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.545665 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.546324 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:58.546736 413 server_base.cc:1047] running on GCE node
I20250904 07:53:58.546986 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:58.547050 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:58.547106 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972438547079 us; error 0 us; skew 500 ppm
I20250904 07:53:58.547226 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:58.548327 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:53:58.548498 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:58.548565 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:58.550177 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:58.551254 7905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:58.551580 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:58.551698 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:58.551847 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:58.563346 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:58.563835 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:58.564356 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:58.567575 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:53:58.567690 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:53:58.569579 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:53:58.569660 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20250904 07:53:58.569729 7912 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:53:58.583977 7969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:53:58.583974 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:53:58.585455 7912 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:58.585925 7912 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:53:58.586274 7912 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250904 07:53:58.586990 7912 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.587152 7912 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:53:58.587460 7912 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 7.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.589388 7912 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250904 07:53:58.593482 7970 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:58.593649 7970 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:58.594172 7970 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:58.595345 6069 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:53:58.596930 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58674
I20250904 07:53:58.598551 7970 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:58.599221 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:53:58.607944 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:53:58.608186 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.608366 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 LEADER]: Raft consensus shutting down.
I20250904 07:53:58.608695 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.609414 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:53:58.609553 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250904 07:53:58.609762 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250904 07:53:58.614156 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:53:58.617200 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:53:58.619678 7978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:53:58.619768 7979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:58.620693 413 server_base.cc:1047] running on GCE node
W20250904 07:53:58.620919 7981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:53:58.621274 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:53:58.621363 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:53:58.621438 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972438621404 us; error 0 us; skew 500 ppm
I20250904 07:53:58.621603 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:53:58.622718 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:53:58.622926 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:53:58.623023 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:53:58.624624 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:53:58.625910 7986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:53:58.626231 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:53:58.626343 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:53:58.626479 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:53:58.636204 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:53:58.636700 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:53:58.637236 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:53:58.642489 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:53:58.642585 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:53:58.644459 7993 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:58.645987 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:53:58.646071 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20250904 07:53:58.661886 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:53:58.661976 8050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:53:58.662147 7993 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=1 applied=17 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:53:58.662585 7993 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:58.662885 7993 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20250904 07:53:58.664892 7993 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.665207 7993 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:58.665548 7993 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 8.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:58.667018 7993 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:53:58.667310 7993 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:53:58.670689 8051 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:53:58.670835 8051 heartbeater.cc:461] Registering TS with master...
I20250904 07:53:58.671263 8051 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:53:58.672466 6069 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:58.673853 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58678
I20250904 07:53:58.674870 8051 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:53:58.680135 7993 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250904 07:53:58.682493 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:58.693147 7993 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:53:58.693420 7993 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.001s
I20250904 07:53:58.694061 7993 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:53:58.694242 7993 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:53:58.694458 7993 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 7.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
W20250904 07:53:58.694618 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:58.694947 7993 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:53:58.704468 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.710815 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.721288 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.734512 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.745152 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.754597 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.767532 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.779062 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.795643 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.810873 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.826489 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.843108 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.862260 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.884169 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.905728 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:58.928788 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.951484 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:58.975281 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.002393 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.029143 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.057922 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.087149 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.118042 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.149857 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.184060 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.217837 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.252656 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.289261 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.325217 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.362157 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.400430 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.440373 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.477268 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.520669 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.562592 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.606536 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.653136 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.699321 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:53:59.726389 8057 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:53:59.726585 8057 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:59.727449 8057 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:59.733563 7782 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:53:59.733608 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:59.733822 8026 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 8.
I20250904 07:53:59.734220 7827 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:53:59.734440 8057 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250904 07:53:59.734508 8057 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:53:59.734555 8057 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:53:59.736548 8057 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:59.737080 8057 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 9 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:53:59.737428 7782 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:53:59.737509 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:53:59.737648 8026 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:53:59.739496 8026 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 9.
I20250904 07:53:59.739874 7827 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:53:59.740124 8057 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Leader election won for term 9
I20250904 07:53:59.740463 8057 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:53:59.740723 8057 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 8.17, Last appended by leader: 17, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
W20250904 07:53:59.742573 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:53:59.743062 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 8 to 9. New cstate: current_term: 9 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
W20250904 07:53:59.791018 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.839948 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:53:59.886027 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:53:59.936267 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:53:59.955422 8068 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:53:59.955588 8068 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:53:59.956352 8068 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:53:59.962811 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:53:59.962831 7866 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:53:59.963073 8026 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 in current term 9: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:53:59.963454 7743 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:53:59.963718 8068 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:53:59.965819 8068 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250904 07:53:59.985882 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.036576 7925 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59396: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.091768 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:00.099749 8072 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:00.099905 8072 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.100723 8072 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:00.106061 7866 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 8 candidate_status { last_received { term: 7 index: 13 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:00.106066 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 8 candidate_status { last_received { term: 7 index: 13 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:00.106287 7866 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 7.
I20250904 07:54:00.106318 8026 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 7.
I20250904 07:54:00.106660 7909 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:00.106995 8072 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250904 07:54:00.107115 8072 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:00.107211 8072 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 7 FOLLOWER]: Advancing to term 8
I20250904 07:54:00.109248 8072 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.109827 8072 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 8 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:00.110110 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 8 candidate_status { last_received { term: 7 index: 13 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:00.110226 7866 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 8 candidate_status { last_received { term: 7 index: 13 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:00.110284 8026 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 7 FOLLOWER]: Advancing to term 8
I20250904 07:54:00.110424 7866 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 7 FOLLOWER]: Advancing to term 8
I20250904 07:54:00.112360 8026 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 8.
I20250904 07:54:00.112356 7866 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 8.
I20250904 07:54:00.112725 7909 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:00.113085 8072 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Leader election won for term 8
I20250904 07:54:00.113457 8072 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 LEADER]: Becoming Leader. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Running, Role: LEADER
I20250904 07:54:00.113749 8072 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 7.13, Last appended by leader: 13, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.115746 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c reported cstate change: term changed from 7 to 8, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68). New cstate: current_term: 8 leader_uuid: "d67ae74f80964d33aaf83b8e54ed263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: HEALTHY } } }
W20250904 07:54:00.142392 7846 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57462: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:00.169183 7782 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 8 index: 17. Preceding OpId from leader: term: 9 index: 18. (index mismatch)
I20250904 07:54:00.169718 8057 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250904 07:54:00.173677 8026 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 8 index: 17. Preceding OpId from leader: term: 9 index: 18. (index mismatch)
I20250904 07:54:00.174261 8057 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250904 07:54:00.199744 8026 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 7 index: 13. Preceding OpId from leader: term: 8 index: 15. (index mismatch)
I20250904 07:54:00.199766 7866 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 7 index: 11. Preceding OpId from leader: term: 8 index: 15. (index mismatch)
I20250904 07:54:00.200244 8072 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250904 07:54:00.200470 8075 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20250904 07:54:00.214531 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:54:00.229856 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:00.230118 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:00.230300 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:00.230489 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:00.244904 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:54:00.247825 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:00.250330 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.250360 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.250453 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:00.251152 413 server_base.cc:1047] running on GCE node
I20250904 07:54:00.251363 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:00.251436 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:00.251477 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972440251463 us; error 0 us; skew 500 ppm
I20250904 07:54:00.251601 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:00.252650 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:54:00.252904 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:00.253007 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:00.254688 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250904 07:54:00.255769 8100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:00.256096 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:00.256227 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:00.256366 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:00.274343 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:00.274761 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:00.275398 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:00.276567 8107 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:00.278967 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:00.279064 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:00.279162 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:00.280808 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:00.280915 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:00.280961 8107 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:54:00.295545 8107 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:00.295877 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:54:00.295948 8164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:54:00.296044 8107 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:54:00.296295 8107 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20250904 07:54:00.297111 8107 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:00.297372 8107 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:54:00.297732 8107 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 9.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:00.299015 8107 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:54:00.303231 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:54:00.305588 8165 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:00.306107 8165 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:00.306742 8165 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:00.308079 6069 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:54:00.309666 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58682
I20250904 07:54:00.311239 8165 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:00.312000 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:00.312978 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:00.313174 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:00.313339 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:00.314090 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:00.314201 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 LEADER]: Raft consensus shutting down.
I20250904 07:54:00.314445 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:00.328625 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:54:00.331871 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:00.334263 8170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.334287 8171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.334713 8173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:00.334915 413 server_base.cc:1047] running on GCE node
I20250904 07:54:00.335146 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:00.335216 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:00.335289 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972440335256 us; error 0 us; skew 500 ppm
I20250904 07:54:00.335454 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:00.336541 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:54:00.336753 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:00.336845 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:00.338693 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250904 07:54:00.339885 8178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:00.340215 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:00.340322 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:00.340451 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:00.351420 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:00.351889 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:00.352341 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:00.358043 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:00.358140 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:54:00.360176 8185 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:00.361583 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:00.361675 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.002s
I20250904 07:54:00.374962 8185 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:00.375355 8185 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:00.375586 8185 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250904 07:54:00.376252 8185 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:00.376412 8185 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:00.376688 8185 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 9.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:00.376765 8242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:54:00.376741 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:54:00.377360 8185 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250904 07:54:00.377624 8185 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:00.384106 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:54:00.391840 8244 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:00.392048 8244 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:00.392437 8244 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:00.393769 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:00.395826 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58688
I20250904 07:54:00.399760 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:00.400033 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:00.400260 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 LEADER]: Raft consensus shutting down.
I20250904 07:54:00.400571 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:00.403380 8185 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:00.403872 8185 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:00.404181 8185 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.004s
I20250904 07:54:00.405126 8185 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.405316 8185 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:00.405586 8185 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 8.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.405961 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:54:00.405995 8244 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:00.406100 8185 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250904 07:54:00.409997 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:00.412374 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.412374 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:00.414304 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:00.414587 413 server_base.cc:1047] running on GCE node
I20250904 07:54:00.414829 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:00.414882 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:00.414944 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972440414940 us; error 0 us; skew 500 ppm
I20250904 07:54:00.415081 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:00.416141 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:54:00.416287 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:00.416325 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:00.417796 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:00.418782 8257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:00.419049 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:00.419126 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:00.419221 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:00.428304 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:00.428674 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:00.429152 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:00.432179 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:00.432261 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20250904 07:54:00.434059 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:00.434118 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:00.434190 8264 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:54:00.448066 8264 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:00.448472 8264 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:54:00.448704 8264 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20250904 07:54:00.449249 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:54:00.449301 8321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:54:00.449388 8264 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.449525 8264 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:54:00.449739 8264 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 8.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:00.451038 8264 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20250904 07:54:00.456434 8322 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:00.456544 8322 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:00.456844 8322 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
W20250904 07:54:00.456977 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:00.457917 6069 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:00.458918 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58702
I20250904 07:54:00.459805 8322 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:00.460409 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.469067 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.475581 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.482868 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.489126 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.500640 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.513165 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.523720 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.538955 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.553685 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.568540 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.583662 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.598222 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.615931 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.636109 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.657845 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.677662 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.702804 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.724516 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.749364 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.774662 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.802037 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.830924 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.858911 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.886543 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.916147 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:00.948227 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:00.980036 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.015058 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.049454 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:01.082352 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.116218 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.153699 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:01.193735 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.234390 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.277076 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:01.317736 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.360957 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.407281 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:01.451136 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.496163 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:01.506038 8328 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:01.506238 8328 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:01.506989 8328 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:01.512531 8140 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:01.512576 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:01.512800 8140 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 9.
I20250904 07:54:01.513216 8179 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:01.513459 8328 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250904 07:54:01.513572 8328 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:01.513646 8328 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Advancing to term 10
I20250904 07:54:01.515652 8328 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:01.516175 8328 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:01.516546 8140 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:54:01.516554 8026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:01.516688 8140 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 9 FOLLOWER]: Advancing to term 10
I20250904 07:54:01.518589 8140 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 10.
I20250904 07:54:01.518906 8179 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:01.519184 8328 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader election won for term 10
I20250904 07:54:01.519537 8328 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:54:01.519840 8328 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 9.18, Last appended by leader: 18, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:01.522097 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 9 to 10. New cstate: current_term: 10 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
W20250904 07:54:01.542723 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:01.587810 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.634683 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:01.676904 8339 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:01.677114 8339 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:01.677939 8339 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:01.684967 8218 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:01.684968 8140 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:01.685297 8140 raft_consensus.cc:2391] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 10: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:54:01.685638 7987 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40; no voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7
I20250904 07:54:01.685947 8339 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Advancing to term 10
W20250904 07:54:01.686518 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:01.688251 8339 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250904 07:54:01.738258 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.787283 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:01.840834 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:01.884012 8339 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:54:01.884248 8339 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:01.885042 8339 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:01.885483 8218 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:01.885696 8218 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 8.
I20250904 07:54:01.886129 7990 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:01.886399 8339 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250904 07:54:01.886494 8339 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:54:01.886621 8339 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:54:01.889187 8339 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:01.889783 8339 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 9 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:01.890045 8218 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:01.890223 8218 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:54:01.890841 8297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:01.890947 8296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 9 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:01.891062 8297 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 8.
W20250904 07:54:01.892575 8006 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48400: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:01.892990 8218 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 9.
I20250904 07:54:01.893368 7990 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:01.893640 8339 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Leader election won for term 9
I20250904 07:54:01.894008 8339 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:01.894295 8339 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 8.17, Last appended by leader: 17, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:01.896411 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 8 to 9, leader changed from d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 9 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250904 07:54:01.944736 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:01.958351 8140 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 9 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250904 07:54:01.958956 8328 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250904 07:54:01.962987 8026 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 9 index: 18. Preceding OpId from leader: term: 10 index: 19. (index mismatch)
I20250904 07:54:01.963639 8328 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250904 07:54:02.001444 8296 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 8 FOLLOWER]: Advancing to term 9
I20250904 07:54:02.001535 8218 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 8 index: 16. Preceding OpId from leader: term: 9 index: 19. (index mismatch)
I20250904 07:54:02.002095 8344 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20250904 07:54:02.004225 8296 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 9 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 8 index: 17. Preceding OpId from leader: term: 9 index: 19. (index mismatch)
I20250904 07:54:02.004918 8344 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250904 07:54:02.007222 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:54:02.008072 8348 mvcc.cc:204] Tried to move back new op lower bound from 7196559122433159168 to 7196559122001072128. Current Snapshot: MvccSnapshot[applied={T|T < 7196559122433159168}]
I20250904 07:54:02.018182 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:02.018427 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:02.018680 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 LEADER]: Raft consensus shutting down.
I20250904 07:54:02.019376 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:02.020009 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:02.020179 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:02.020319 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:02.034579 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:54:02.037787 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:02.040117 8360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:02.040169 8361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:02.040748 8363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:02.040870 413 server_base.cc:1047] running on GCE node
I20250904 07:54:02.041158 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:02.041241 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:02.041301 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972442041282 us; error 0 us; skew 500 ppm
I20250904 07:54:02.041553 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:02.042812 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:54:02.043053 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:02.043128 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:02.044762 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:02.046051 8368 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:02.046416 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:02.046522 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:02.046623 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:02.059217 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:02.059623 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:02.060074 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:02.061162 8375 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:54:02.065728 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:02.065829 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20250904 07:54:02.065958 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:54:02.067868 8375 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:02.069567 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:02.069639 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.002s
I20250904 07:54:02.083010 8375 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=1 applied=19 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:02.083452 8375 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:02.083745 8375 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250904 07:54:02.084482 8375 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:02.084692 8375 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:02.084995 8375 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:02.085430 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:54:02.085505 8432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:54:02.085755 8375 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20250904 07:54:02.086005 8375 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:02.095577 8434 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:02.095743 8434 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:02.096153 8434 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:02.097426 6068 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:02.099225 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58708
W20250904 07:54:02.104684 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:02.107929 8375 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:02.108269 8375 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:02.108532 8375 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
W20250904 07:54:02.109186 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:02.109253 8375 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:02.109452 8375 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:02.109710 8375 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 9.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:02.110051 8434 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:02.110176 8375 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20250904 07:54:02.114897 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.123214 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.131621 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.141624 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.154115 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.165707 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.178474 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.193033 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.207643 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.222654 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.238292 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.257947 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.279034 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.299849 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.319609 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.344813 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.365365 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.390883 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.418378 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.442106 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.467833 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.496903 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.527503 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.557211 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.590354 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.623154 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.654058 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.691565 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.726475 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.763504 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.809759 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.848661 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.887542 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:02.930900 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:02.970487 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.011178 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.056056 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.100845 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.143543 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.188704 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.233541 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.280376 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.330790 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.379541 8198 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57492: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.432245 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.486877 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:03.508128 8328 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:03.508319 8328 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.509023 8328 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:03.509567 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:03.509801 8408 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 9.
I20250904 07:54:03.510200 8179 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:03.510471 8328 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250904 07:54:03.510623 8328 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:03.510705 8328 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 9 FOLLOWER]: Advancing to term 10
I20250904 07:54:03.511188 8444 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:03.511343 8444 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.512069 8444 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:03.513953 8328 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.514967 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:03.515144 8408 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 9 FOLLOWER]: Advancing to term 10
I20250904 07:54:03.518744 8408 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 10.
I20250904 07:54:03.519268 8179 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:03.519553 8328 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 10 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:03.519750 8328 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader election won for term 10
I20250904 07:54:03.519917 8328 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:54:03.520207 8328 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 9.19, Last appended by leader: 19, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.529547 8296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:03.529609 8297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:03.529732 8296 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 9.
I20250904 07:54:03.530027 6068 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 9 to 10, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 10 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:03.533771 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:03.533989 8408 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 10: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:54:03.534832 8218 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:03.535302 8261 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:03.535497 8444 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 9 FOLLOWER]: Advancing to term 10
I20250904 07:54:03.537760 8444 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250904 07:54:03.539031 8297 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 10 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 9 index: 19. Preceding OpId from leader: term: 10 index: 21. (index mismatch)
I20250904 07:54:03.539350 8408 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 9 index: 19. Preceding OpId from leader: term: 10 index: 21. (index mismatch)
I20250904 07:54:03.539454 8449 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20250904 07:54:03.539767 8328 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250904 07:54:03.545109 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:54:03.555112 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:03.555616 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:03.555826 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:03.556030 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:03.570080 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:54:03.573150 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:03.575320 8465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:03.575372 8466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:03.575796 8468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:03.576556 413 server_base.cc:1047] running on GCE node
I20250904 07:54:03.576826 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:03.576938 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:03.576970 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972443576967 us; error 0 us; skew 500 ppm
I20250904 07:54:03.577108 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:03.578204 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:54:03.578389 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:03.578436 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:03.580004 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:03.581121 8473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:03.581420 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:03.581502 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:03.581605 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:03.591789 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:03.592196 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:03.592636 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:03.593647 8480 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:03.595840 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:03.595909 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:03.595995 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:03.597865 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:03.597923 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:03.598004 8480 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
W20250904 07:54:03.609069 8179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.66:32779: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20250904 07:54:03.610275 8179 consensus_peers.cc:489] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 -> Peer d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779): Couldn't send request to peer d888acb801154571b952c5dba8b792d7. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:32779: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 07:54:03.611285 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:54:03.611302 8538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:54:03.619015 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:54:03.619328 8480 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:03.619753 8480 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:54:03.620057 8480 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20250904 07:54:03.620975 8480 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:03.621156 8480 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:54:03.621394 8480 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 10.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:03.622437 8480 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20250904 07:54:03.623770 8539 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:03.623920 8539 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:03.624310 8539 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:03.626078 6069 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:54:03.628115 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58720
I20250904 07:54:03.628485 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:03.628720 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:03.628957 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 LEADER]: Raft consensus shutting down.
I20250904 07:54:03.629256 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:03.629309 8539 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:03.629938 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:03.630082 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 LEADER]: Raft consensus shutting down.
I20250904 07:54:03.630352 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:03.645476 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:54:03.648816 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:03.650980 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:03.651114 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:03.651465 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:03.651921 413 server_base.cc:1047] running on GCE node
I20250904 07:54:03.652210 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:03.652264 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:03.652307 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972443652304 us; error 0 us; skew 500 ppm
I20250904 07:54:03.652442 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:03.653615 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:54:03.653796 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:03.653843 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:03.655426 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:03.656636 8554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:03.656980 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:03.657075 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:03.657181 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:03.667853 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:03.668275 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:03.668730 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:03.674161 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:03.674242 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20250904 07:54:03.676157 8561 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:03.677791 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:03.677861 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250904 07:54:03.691713 8561 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:03.692099 8561 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:03.692373 8561 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250904 07:54:03.693153 8561 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:03.693728 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:54:03.693742 8618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:54:03.693298 8561 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:03.694763 8561 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 10.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:03.707716 8561 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20250904 07:54:03.707957 8561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:03.711356 8619 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:03.711493 8619 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:03.711947 8619 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:03.713152 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:03.714355 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58726
I20250904 07:54:03.715524 8619 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:03.716145 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.720666 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:03.724439 8561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=21 overwritten=0 applied=21 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:03.724788 8561 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:03.725018 8561 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250904 07:54:03.725613 8561 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.725749 8561 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:03.725970 8561 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 10.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:03.726352 8561 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:54:03.729743 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.736218 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.742661 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.753547 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.763151 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.776552 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.792393 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.807881 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.821350 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.836303 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.852893 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.872527 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.893404 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.912987 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.935626 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:03.956642 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:03.977366 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.001044 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.026923 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.051539 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.080421 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.107621 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.137339 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.165143 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.195252 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.227985 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.260769 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.294901 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.329677 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.363489 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.400748 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.436578 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.477578 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.517850 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.560019 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.599845 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.643981 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.689896 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.733570 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.779049 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.828133 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.877251 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:04.925493 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:04.977531 8388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48468: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:04.995189 8626 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:04.995365 8626 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:04.996080 8626 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:05.005883 8631 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:05.006691 8631 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.015807 8631 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:05.018217 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:05.018838 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:05.022397 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 11 candidate_status { last_received { term: 10 index: 19 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:05.022648 8594 raft_consensus.cc:2408] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d888acb801154571b952c5dba8b792d7 for term 11 because replica has last-logged OpId of term: 10 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 19.
I20250904 07:54:05.023041 8477 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:05.023341 8626 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 07:54:05.024478 8297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:05.024921 8558 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:05.025158 8631 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250904 07:54:05.029103 8277 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59402: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:05.044646 8636 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:05.044775 8636 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.045399 8636 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:05.051118 8297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:05.051126 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:05.051316 8594 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 10.
I20250904 07:54:05.051649 8372 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:05.051934 8636 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250904 07:54:05.052037 8636 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:05.052138 8636 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Advancing to term 11
I20250904 07:54:05.054194 8636 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.054666 8636 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:05.055013 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:05.055088 8297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:05.055161 8594 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Advancing to term 11
I20250904 07:54:05.057148 8594 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 11.
I20250904 07:54:05.057458 8372 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:05.057708 8636 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Leader election won for term 11
I20250904 07:54:05.058068 8636 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:05.058346 8636 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 10.21, Last appended by leader: 21, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.060462 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 10 to 11, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 11 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:05.063951 8645 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:05.064136 8645 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.064631 8645 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:05.064965 8408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:05.069414 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:05.069629 8594 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 11: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:54:05.069975 8261 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:05.071357 8645 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 10 FOLLOWER]: Advancing to term 11
I20250904 07:54:05.073275 8645 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250904 07:54:05.084589 8297 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 10 index: 21. Preceding OpId from leader: term: 11 index: 23. (index mismatch)
I20250904 07:54:05.084645 8594 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 10 index: 21. Preceding OpId from leader: term: 11 index: 23. (index mismatch)
I20250904 07:54:05.085078 8639 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250904 07:54:05.085256 8636 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250904 07:54:05.093786 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:54:05.111186 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:05.111380 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:05.111533 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:05.111763 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:05.118708 8652 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:05.118829 8652 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:05.119354 8652 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:05.119815 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:05.119999 8594 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 10.
I20250904 07:54:05.120738 8372 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:05.120982 8652 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250904 07:54:05.121052 8652 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:05.121112 8652 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 10 FOLLOWER]: Advancing to term 11
I20250904 07:54:05.123730 8652 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:05.124217 8652 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:05.124579 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:05.124699 8513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:05.124738 8594 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 10 FOLLOWER]: Advancing to term 11
I20250904 07:54:05.124802 8512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:54:05.124838 8513 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 10.
I20250904 07:54:05.125744 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:54:05.128334 8594 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 11.
I20250904 07:54:05.128686 8372 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:54:05.128916 8652 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Leader election won for term 11
I20250904 07:54:05.129032 8652 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:05.129232 8652 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 10.20, Last appended by leader: 20, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:05.129442 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:05.131799 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:05.131882 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:05.131839 6068 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 10 to 11, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 11 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
W20250904 07:54:05.134294 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:05.134652 413 server_base.cc:1047] running on GCE node
I20250904 07:54:05.134862 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:05.134927 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:05.134963 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972445134960 us; error 0 us; skew 500 ppm
I20250904 07:54:05.135106 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:05.136271 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:54:05.136449 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:05.136494 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:05.138020 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:05.139060 8666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:05.139328 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:05.139417 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:05.139503 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:05.156989 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:05.157356 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:05.157784 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:05.158833 8673 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:05.161108 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:05.161178 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:05.161266 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:05.162905 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:05.162959 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:54:05.163062 8673 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:54:05.177305 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:54:05.177325 8730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:54:05.180826 8673 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=23 ignored=0} inserts{seen=13 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:05.181262 8673 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:54:05.181556 8673 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.002s
I20250904 07:54:05.182384 8673 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.182557 8673 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:54:05.182821 8673 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 11.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.183718 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:54:05.191260 8673 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20250904 07:54:05.192540 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:05.192793 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:05.193010 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 LEADER]: Raft consensus shutting down.
I20250904 07:54:05.193257 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:05.193953 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:05.194087 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 LEADER]: Raft consensus shutting down.
I20250904 07:54:05.194317 413 pending_rounds.cc:70] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Trying to abort 1 pending ops.
I20250904 07:54:05.194365 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Aborting op as it isn't in flight: id { term: 11 index: 21 } timestamp: 7196559135251390464 op_type: NO_OP noop_request { }
I20250904 07:54:05.194474 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:05.194573 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:05.198063 8731 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:05.198196 8731 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:05.198571 8731 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
W20250904 07:54:05.200367 8670 connection.cc:441] server connection from 127.0.0.1:59462 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59462 (request call id 0) could send its response
I20250904 07:54:05.201489 6068 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:05.202471 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58730
I20250904 07:54:05.203434 8731 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:05.212221 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:54:05.215299 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:05.217486 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:05.217562 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:05.218029 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:05.218335 413 server_base.cc:1047] running on GCE node
I20250904 07:54:05.218537 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:05.218590 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:05.218637 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972445218633 us; error 0 us; skew 500 ppm
I20250904 07:54:05.218770 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:05.219794 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:54:05.219977 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:05.220022 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:05.221625 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:54:05.222749 8748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:05.223038 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:05.223131 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:05.223227 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:05.238107 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:05.238482 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:05.238921 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:05.244004 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:05.244076 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:54:05.245903 8755 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:05.247373 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:05.247433 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20250904 07:54:05.261957 8755 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=1 applied=20 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250904 07:54:05.262172 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:54:05.262192 8812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:54:05.262360 8755 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:05.262599 8755 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250904 07:54:05.263338 8755 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:05.263587 8755 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:05.263862 8755 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 11.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
W20250904 07:54:05.264683 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:05.264816 8755 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20250904 07:54:05.265066 8755 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:05.269316 8813 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:05.269433 8813 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:05.269722 8813 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:05.270750 6068 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:05.272184 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58734
I20250904 07:54:05.273238 8813 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:05.273677 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.283001 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:05.283205 8755 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=14 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:05.283573 8755 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:05.283756 8755 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20250904 07:54:05.284231 8755 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.284356 8755 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:05.284546 8755 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 11.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:05.284897 8755 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
W20250904 07:54:05.288368 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.294531 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.302752 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.313076 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.325270 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.335592 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.348172 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.363615 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.377035 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.394735 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.412073 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.429416 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.450949 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.472450 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.493866 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.515695 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.538945 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.563391 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.589061 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.616292 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.643683 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.673481 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.702888 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.734504 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.764802 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.794572 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.828469 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.863068 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.898802 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.934643 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:05.973935 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.012777 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.049727 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.089074 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.127720 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.171689 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.215492 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.258196 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.303838 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.348891 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.395788 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.444598 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.495740 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.544535 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
W20250904 07:54:06.595520 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:06.635747 8820 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:06.635905 8820 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:06.636659 8820 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
W20250904 07:54:06.649856 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:06.655042 8513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:06.655267 8513 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 10.
I20250904 07:54:06.655436 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:06.655596 8594 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 11.
I20250904 07:54:06.655637 8749 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:54:06.655865 8820 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250904 07:54:06.655952 8820 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:06.656021 8820 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.658077 8820 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:06.658560 8820 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 12 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:06.658921 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:06.658921 8513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:54:06.659097 8594 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.659097 8513 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 10 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.661635 8594 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 12.
I20250904 07:54:06.661635 8513 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 12.
I20250904 07:54:06.661996 8749 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7; no voters:
I20250904 07:54:06.662214 8820 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Leader election won for term 12
I20250904 07:54:06.662562 8820 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:06.662961 8820 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 11.21, Last appended by leader: 21, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:06.665108 6068 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 11 to 12. New cstate: current_term: 12 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:06.684096 8831 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:06.684238 8831 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:06.684829 8831 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:06.699599 8836 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:06.699766 8836 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
W20250904 07:54:06.703522 8493 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52408: Illegal state: replica d888acb801154571b952c5dba8b792d7 is not leader of this config: current role FOLLOWER
I20250904 07:54:06.705161 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:06.705379 8788 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 11.
I20250904 07:54:06.705865 8555 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:06.706184 8831 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250904 07:54:06.706368 8831 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:06.706494 8831 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 11 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.707759 8836 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:06.710668 8831 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:06.711910 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:06.712011 8831 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 12 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:06.712093 8788 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 11 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.715925 8706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:06.716059 8787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:06.716087 8705 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:06.716118 8706 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 11.
I20250904 07:54:06.716464 8788 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 12.
I20250904 07:54:06.716825 8555 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:06.717130 8831 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Leader election won for term 12
I20250904 07:54:06.717573 8831 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:54:06.717886 8831 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 11.24, Last appended by leader: 24, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:06.719785 8594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:06.720252 8670 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:06.720285 6068 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 11 to 12, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 12 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:06.720518 8836 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250904 07:54:06.756507 8574 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57520: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:06.812989 8513 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 10 index: 19. Preceding OpId from leader: term: 12 index: 23. (index mismatch)
I20250904 07:54:06.813002 8594 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 10 index: 20. Preceding OpId from leader: term: 12 index: 23. (index mismatch)
I20250904 07:54:06.813695 8825 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.000s
I20250904 07:54:06.813970 8820 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20250904 07:54:06.822324 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
W20250904 07:54:06.834208 8749 consensus_peers.cc:489] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 -> Peer d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779): Couldn't send request to peer d888acb801154571b952c5dba8b792d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:54:06.845270 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:06.845499 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:06.845690 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:06.845860 413 pending_rounds.cc:70] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Trying to abort 3 pending ops.
I20250904 07:54:06.845894 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Aborting op as it isn't in flight: id { term: 12 index: 23 } timestamp: 7196559142140989440 op_type: WRITE_OP write_request { tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" write_default_value: "90\000\000" immutable: false } } row_operations { rows: "\001\017\000\031\000\000\0002\000\000\000\000\000\000\000\000\000\000\000\010\000\000\000\000\000\000\000K\000\000\000""\001\017\000\031\000\000\0002\000\000\000\000\000\000\000\000\000\000\000\010\000\000\000\000\000\000\000K\000\000\000" indirect_data: "hello 25""hello 25" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7196559135456239616 authz_token { token_data: "\010\267\215\345\305\006\"3\n\005slave\022*\n e7ec7b5e119243009ba67e3375146cff\020\001\030\001 \001(\001" signature: "+\221\371\231E\355\'\223z9\'\374\242;\\\345/\310O\357\262\202lnv\355\366\343\303\377\261]j\235\256W3\247\005^@\343\274FMzU\343\026\274\265\277\244\300?\242A\264\"\\\275Y\347\254""+\221\371\231E\355\'\223z9\'\374\242;\\\345/\310O\357\262\202lnv\355\366\343\303\377\261]j\235\256W3\247\005^@\343\274FMzU\343\026\274\265\277\244\300?\242A\264\"\\\275Y\347\254" signing_key_seq_num: 0 } } request_id { client_id: "8f2ade9ba4334c76894179e301c67080" seq_no: 25 first_incomplete_seq_no: 25 attempt_no: 51 }
I20250904 07:54:06.846253 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Aborting op as it isn't in flight: id { term: 12 index: 22 } timestamp: 7196559141533638656 op_type: NO_OP noop_request { }
I20250904 07:54:06.846329 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:06.846385 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Aborting op as it isn't in flight: id { term: 11 index: 21 } timestamp: 7196559135251390464 op_type: NO_OP noop_request { }
I20250904 07:54:06.846424 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:06.846472 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:06.850281 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:54:06.853236 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:06.855541 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:06.855628 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:06.856302 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:06.856315 413 server_base.cc:1047] running on GCE node
I20250904 07:54:06.856659 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:06.856717 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:06.856745 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972446856741 us; error 0 us; skew 500 ppm
I20250904 07:54:06.856917 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:06.858001 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:54:06.858201 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:06.858264 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:06.859820 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:06.860946 8860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:06.861224 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:06.861307 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:06.861410 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:06.892606 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:06.893146 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:06.893709 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:06.894726 8867 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:06.897140 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:06.897222 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:06.897318 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:06.899190 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:06.899248 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:54:06.899348 8867 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:54:06.914247 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:54:06.914259 8924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:54:06.914867 8867 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=20 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250904 07:54:06.915305 8867 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:54:06.915699 8867 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.000s
I20250904 07:54:06.916702 8867 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
W20250904 07:54:06.917202 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:06.919970 8867 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:54:06.920353 8867 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 12.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:06.921125 8867 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20250904 07:54:06.924335 8706 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 11 FOLLOWER]: Advancing to term 12
I20250904 07:54:06.924434 8788 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 11 index: 24. Preceding OpId from leader: term: 12 index: 26. (index mismatch)
I20250904 07:54:06.924971 8831 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250904 07:54:06.927254 8925 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:06.927381 8925 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:06.927448 8706 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 11 index: 24. Preceding OpId from leader: term: 12 index: 26. (index mismatch)
I20250904 07:54:06.927728 8925 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:06.928418 8831 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20250904 07:54:06.929080 6069 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:54:06.930608 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58750
I20250904 07:54:06.931295 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:54:06.931946 8925 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:06.940346 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:06.940570 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:06.940738 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 LEADER]: Raft consensus shutting down.
I20250904 07:54:06.941027 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:06.941881 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:06.942023 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:06.942170 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:06.948189 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:54:06.951141 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:06.953188 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:06.953207 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:06.953799 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:06.953943 413 server_base.cc:1047] running on GCE node
I20250904 07:54:06.954221 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:06.954272 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:06.954294 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972446954292 us; error 0 us; skew 500 ppm
I20250904 07:54:06.954380 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:06.955405 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:54:06.955591 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:06.955631 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:06.957432 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:06.958560 8944 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:06.958839 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:06.958941 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:06.959033 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:06.976118 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:06.976505 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:06.976984 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:06.982074 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:06.982146 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:54:06.984006 8951 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:06.985549 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:06.985610 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20250904 07:54:07.000144 8951 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=23 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:07.000388 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:54:07.000409 9008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:54:07.000509 8951 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:07.000726 8951 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20250904 07:54:07.002136 8951 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:07.002743 8951 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:07.003023 8951 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 12.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:07.003717 8951 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.003s user 0.002s sys 0.004s
I20250904 07:54:07.003988 8951 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:07.007740 9009 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:07.007863 9009 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:07.008189 9009 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:07.009300 6069 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:07.010588 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58762
I20250904 07:54:07.011523 9009 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:07.014803 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.022071 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:07.024581 8951 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:07.024933 8951 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:07.025130 8951 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.011s
I20250904 07:54:07.025631 8951 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:07.025750 8951 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:07.025938 8951 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 12.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:07.026291 8951 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:54:07.028643 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.036960 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.047197 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.059758 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.068971 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.079232 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.094772 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.109107 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.121554 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.140365 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.157717 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.175132 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.195930 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.216372 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.234946 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.255963 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.277521 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.303177 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.327080 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.352681 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.377470 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.406728 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.436403 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.467936 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.501976 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.534606 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.569401 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.606392 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.640187 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.678030 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.714079 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.749764 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.786603 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.826679 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.865481 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.906215 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:07.950081 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:07.993083 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.039848 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.089464 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.136400 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.186208 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.235256 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.283035 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.331862 8686 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59492: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.386245 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:08.422575 9025 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:08.422753 9025 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.423482 9025 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:08.430092 9019 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:08.430544 9019 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.433126 9019 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:08.433504 9030 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:08.433531 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:08.433677 9030 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.433727 8984 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 12.
I20250904 07:54:08.434217 9030 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:08.434532 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:08.434690 8788 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 12.
I20250904 07:54:08.435168 8667 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:08.435599 9030 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Leader pre-election won for term 13
I20250904 07:54:08.435781 9030 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:08.435964 9030 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:08.439942 9030 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.442543 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:08.442747 8788 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:08.443454 9030 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 13 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:08.445883 8788 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 13.
I20250904 07:54:08.446270 8667 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:08.446683 9030 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Leader election won for term 13
I20250904 07:54:08.447126 8752 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:08.447387 9019 raft_consensus.cc:2762] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20250904 07:54:08.448462 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:08.453576 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:08.453908 8705 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:08.453886 8788 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 13: Already voted for candidate d67ae74f80964d33aaf83b8e54ed263c in this term.
I20250904 07:54:08.455143 9030 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 LEADER]: Becoming Leader. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Running, Role: LEADER
I20250904 07:54:08.455740 9030 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 12.26, Last appended by leader: 26, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.459156 6067 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c reported cstate change: term changed from 12 to 13, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68). New cstate: current_term: 13 leader_uuid: "d67ae74f80964d33aaf83b8e54ed263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:08.459750 8705 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:08.460238 8948 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:08.460480 9025 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:08.462471 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:08.462563 8983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 13 candidate_status { last_received { term: 12 index: 26 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:08.462939 9025 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250904 07:54:08.463124 8984 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 13.
I20250904 07:54:08.504386 8984 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 12 index: 26. Preceding OpId from leader: term: 13 index: 28. (index mismatch)
I20250904 07:54:08.504408 8788 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 12 index: 26. Preceding OpId from leader: term: 13 index: 28. (index mismatch)
I20250904 07:54:08.504932 9030 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250904 07:54:08.505079 9035 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250904 07:54:08.510131 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:54:08.520776 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:08.521129 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:08.521337 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 LEADER]: Raft consensus shutting down.
I20250904 07:54:08.521654 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:08.522284 9043 mvcc.cc:204] Tried to move back new op lower bound from 7196559149069680640 to 7196559148877668352. Current Snapshot: MvccSnapshot[applied={T|T < 7196559149069680640}]
I20250904 07:54:08.545100 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:54:08.548290 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:08.550504 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:08.551010 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:08.551038 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:08.551270 413 server_base.cc:1047] running on GCE node
I20250904 07:54:08.551604 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:08.551661 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:08.551683 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972448551681 us; error 0 us; skew 500 ppm
I20250904 07:54:08.551780 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:08.552922 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:54:08.553097 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:08.553139 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:08.554780 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:08.555861 9058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:08.556177 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:08.556278 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:08.556394 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:08.567530 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:08.567911 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:08.568334 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:08.569453 9065 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:08.571851 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:08.571920 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.001s sys 0.000s
I20250904 07:54:08.571975 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:08.573714 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:08.573786 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:08.573868 9065 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:54:08.588430 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:54:08.588433 9122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:54:08.596134 9123 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:08.596271 9123 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:08.596465 9065 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=16 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:08.596601 9123 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:08.596963 9065 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:54:08.597290 9065 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20250904 07:54:08.597810 6067 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:08.598008 9065 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.598196 9065 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:54:08.598403 9065 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 13.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:08.599061 9065 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:08.599241 6067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44128
I20250904 07:54:08.600216 9123 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:08.601354 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.608712 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.617461 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.622897 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.632222 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.642896 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.652254 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.661720 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.675365 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.690855 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.703303 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.717962 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.733568 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.752023 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.769675 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.791150 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.813694 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.838474 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.859874 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.884308 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.911039 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:08.936539 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.964118 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:08.992004 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.020435 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.050174 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.084276 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.115929 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.146495 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.183570 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.220156 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.254776 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.290678 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.330196 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.367882 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.410900 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.451552 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.491439 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.534314 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.577217 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.620965 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.669353 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.716058 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.763829 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.813148 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:09.862938 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.913317 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:09.964793 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:10.010133 9019 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:54:10.010371 9019 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.011237 9019 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:10.011795 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
W20250904 07:54:10.018522 8768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48522: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
I20250904 07:54:10.018517 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:10.018743 9098 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 13.
I20250904 07:54:10.019095 8752 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243
I20250904 07:54:10.019340 9019 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20250904 07:54:10.019450 9019 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:54:10.019546 9019 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:10.022379 9132 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d67ae74f80964d33aaf83b8e54ed263c)
I20250904 07:54:10.022505 9132 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.023147 9019 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.023358 8788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:10.023929 8788 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 14: Already voted for candidate 6a2e703f66a34a598ab3d9646fa5db40 in this term.
I20250904 07:54:10.024186 9132 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:10.025251 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:10.025410 8984 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:10.028271 8984 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 14.
I20250904 07:54:10.028599 8752 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:10.028645 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:10.028775 9098 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:10.030175 9134 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader election won for term 14
I20250904 07:54:10.030562 9097 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 14 candidate_status { last_received { term: 13 index: 28 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:10.031075 8948 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:10.031431 9019 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 election: Requested vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:10.031817 9098 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 14.
I20250904 07:54:10.032109 9132 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20250904 07:54:10.032452 9134 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:10.032783 9134 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 13.28, Last appended by leader: 28, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.035646 6067 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 13 to 14, leader changed from d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 14 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250904 07:54:10.074869 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:10.134977 8984 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 13 index: 28. Preceding OpId from leader: term: 14 index: 30. (index mismatch)
I20250904 07:54:10.135073 9098 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Refusing update from remote peer 6a2e703f66a34a598ab3d9646fa5db40: Log matching property violated. Preceding OpId in replica: term: 13 index: 28. Preceding OpId from leader: term: 14 index: 30. (index mismatch)
I20250904 07:54:10.135461 9134 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250904 07:54:10.135637 9135 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250904 07:54:10.146539 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:54:10.182580 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:10.182884 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:10.183063 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 LEADER]: Raft consensus shutting down.
I20250904 07:54:10.183317 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:10.183975 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:10.184082 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 LEADER]: Raft consensus shutting down.
I20250904 07:54:10.184306 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:10.256987 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:54:10.260412 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:10.262745 9151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:10.263085 9153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:10.263279 413 server_base.cc:1047] running on GCE node
W20250904 07:54:10.263289 9150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:10.263727 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:10.263789 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:10.263833 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972450263829 us; error 0 us; skew 500 ppm
I20250904 07:54:10.263971 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:10.265141 413 webserver.cc:480] Webserver started at http://127.0.103.65:40775/ using document root <none> and password file <none>
I20250904 07:54:10.265318 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:10.265369 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:10.266919 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:10.268177 9158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:10.268461 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:10.268556 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "6a2e703f66a34a598ab3d9646fa5db40"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:10.268663 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:10.291277 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:10.291754 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:10.292246 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:10.293862 9165 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 07:54:10.298200 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:10.298269 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20250904 07:54:10.298322 413 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 07:54:10.300115 9165 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:10.302325 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:10.302397 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250904 07:54:10.315642 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38935
I20250904 07:54:10.315665 9222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38935 every 8 connection(s)
I20250904 07:54:10.324249 9165 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=1 applied=23 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:10.324748 9165 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:10.325050 9165 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.004s
I20250904 07:54:10.325655 9165 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:10.325790 9165 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:10.325985 9165 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 12.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:10.327692 9165 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250904 07:54:10.327939 9165 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap starting.
I20250904 07:54:10.328327 9223 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:10.328445 9223 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:10.328778 9223 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:10.329993 6069 ts_manager.cc:194] Re-registered known tserver with Master: 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
W20250904 07:54:10.330258 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:10.331609 6069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44144
I20250904 07:54:10.332598 9223 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:10.336738 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:10.347442 9165 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=17 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:10.347836 9165 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Bootstrap complete.
I20250904 07:54:10.348040 9165 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20250904 07:54:10.348573 9165 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.348716 9165 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Initialized, Role: FOLLOWER
I20250904 07:54:10.348948 9165 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 14.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:10.349372 9165 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
W20250904 07:54:10.351471 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.358801 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.370294 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.379570 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.389792 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.401260 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.412583 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.426838 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.445410 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.461851 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.477198 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.494907 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.515300 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.536604 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.561496 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.582018 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.607573 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.634665 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.661449 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.690160 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.719482 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.750337 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.778088 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.811134 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.843878 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.875664 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:10.909708 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.945426 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:10.980154 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.016220 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.054838 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.092592 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.134564 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.176067 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:11.197860 9231 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.198025 9231 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.198668 9231 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:11.204295 8900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:11.204295 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:11.204567 9198 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 12.
I20250904 07:54:11.204938 8945 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:54:11.205168 9231 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20250904 07:54:11.205246 9231 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.205324 9231 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:11.207285 9231 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.207751 9231 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:11.208169 8900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:54:11.208169 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:11.208401 9198 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:11.210633 9198 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 13.
I20250904 07:54:11.210937 8945 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d888acb801154571b952c5dba8b792d7
I20250904 07:54:11.211149 9231 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Leader election won for term 13
I20250904 07:54:11.211524 9231 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:54:11.211747 9231 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 12.23, Last appended by leader: 23, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.213681 6069 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 12 to 13, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 13 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } }
W20250904 07:54:11.219308 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.265147 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.310729 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.357262 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.403512 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.450305 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.497985 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.547170 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.594976 8964 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:57580: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
I20250904 07:54:11.627315 9242 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:11.627519 9242 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.628268 9242 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:11.634356 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:11.635843 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:11.636232 9162 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40; no voters: 47e41d546b88492da3428adbff442243, d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:11.636452 9242 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20250904 07:54:11.642642 9247 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.642776 9247 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.643373 9247 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:11.643883 9231 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.644059 9231 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.644937 9231 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
W20250904 07:54:11.645320 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:11.645680 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:11.645915 9198 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 14.
I20250904 07:54:11.646382 8945 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:11.646742 9231 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20250904 07:54:11.646849 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:11.646855 9231 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.647059 9231 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Advancing to term 15
I20250904 07:54:11.647151 9098 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 14.
I20250904 07:54:11.650341 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:11.650367 9197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:11.650472 9231 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.650627 9197 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 14.
I20250904 07:54:11.650645 9198 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 12 index: 23. Preceding OpId from leader: term: 13 index: 24. (index mismatch)
I20250904 07:54:11.650803 8984 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 15: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:54:11.651080 9059 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:11.651350 9236 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250904 07:54:11.651558 9247 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Leader pre-election won for term 15
I20250904 07:54:11.651698 9247 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.651561 9250 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6a2e703f66a34a598ab3d9646fa5db40)
I20250904 07:54:11.651824 9247 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 14 FOLLOWER]: Advancing to term 15
I20250904 07:54:11.651870 9250 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.652218 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c"
I20250904 07:54:11.652519 9231 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 15 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:11.652947 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:11.653148 9198 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Advancing to term 15
I20250904 07:54:11.656792 9247 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.657029 9198 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e41d546b88492da3428adbff442243 in term 15.
I20250904 07:54:11.657325 9098 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 47e41d546b88492da3428adbff442243 in current term 15: Already voted for candidate d67ae74f80964d33aaf83b8e54ed263c in this term.
I20250904 07:54:11.657775 9250 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:11.658998 8945 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:11.660015 9247 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 15 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:11.660034 9231 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Leader election won for term 15
I20250904 07:54:11.661155 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:11.661659 8984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 15 candidate_status { last_received { term: 14 index: 30 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:11.662076 9231 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 LEADER]: Becoming Leader. State: Replica: 47e41d546b88492da3428adbff442243, State: Running, Role: LEADER
I20250904 07:54:11.662411 9231 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 14.30, Last appended by leader: 30, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.662120 9198 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d67ae74f80964d33aaf83b8e54ed263c in current term 15: Already voted for candidate 47e41d546b88492da3428adbff442243 in this term.
I20250904 07:54:11.670756 9197 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:11.673779 8983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:11.674417 8864 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d888acb801154571b952c5dba8b792d7; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:11.674912 9250 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20250904 07:54:11.676468 9062 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:11.677019 9247 raft_consensus.cc:2747] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250904 07:54:11.677326 6069 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 reported cstate change: term changed from 14 to 15, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to 47e41d546b88492da3428adbff442243 (127.0.103.67). New cstate: current_term: 15 leader_uuid: "47e41d546b88492da3428adbff442243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:11.680718 8900 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 12 FOLLOWER]: Advancing to term 13
I20250904 07:54:11.683173 8900 raft_consensus.cc:1273] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 12 index: 23. Preceding OpId from leader: term: 13 index: 24. (index mismatch)
I20250904 07:54:11.683665 9236 consensus_queue.cc:1035] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250904 07:54:11.700565 9098 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 14 index: 30. Preceding OpId from leader: term: 15 index: 32. (index mismatch)
I20250904 07:54:11.700676 9198 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Refusing update from remote peer 47e41d546b88492da3428adbff442243: Log matching property violated. Preceding OpId in replica: term: 14 index: 30. Preceding OpId from leader: term: 15 index: 32. (index mismatch)
I20250904 07:54:11.701113 9255 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250904 07:54:11.701310 9236 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250904 07:54:11.712105 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:54:11.721050 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:11.721261 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:11.721424 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:11.721568 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:11.735541 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:54:11.738654 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:11.740831 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:11.741277 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:11.741453 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:11.741662 413 server_base.cc:1047] running on GCE node
I20250904 07:54:11.741869 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:11.741925 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:11.741971 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972451741967 us; error 0 us; skew 500 ppm
I20250904 07:54:11.742100 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:11.743191 413 webserver.cc:480] Webserver started at http://127.0.103.66:44999/ using document root <none> and password file <none>
I20250904 07:54:11.743378 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:11.743430 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:11.745009 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:11.746054 9274 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:11.746356 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:11.746450 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "d888acb801154571b952c5dba8b792d7"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:11.746551 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:11.773097 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:11.773543 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:11.774060 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:11.775082 9281 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:11.777426 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:11.777509 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:11.777614 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:11.779443 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:11.779500 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250904 07:54:11.779644 9281 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap starting.
I20250904 07:54:11.794586 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:32779
I20250904 07:54:11.794595 9338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:32779 every 8 connection(s)
I20250904 07:54:11.797845 9281 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:11.798311 9281 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Bootstrap complete.
I20250904 07:54:11.798619 9281 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20250904 07:54:11.799414 9281 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.800279 9281 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: d888acb801154571b952c5dba8b792d7, State: Initialized, Role: FOLLOWER
I20250904 07:54:11.800645 9281 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 13.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.801532 9281 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:11.803866 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:54:11.804293 9339 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:11.804406 9339 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:11.804803 9339 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:11.806025 6067 ts_manager.cc:194] Re-registered known tserver with Master: d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779)
I20250904 07:54:11.807657 6067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44148
I20250904 07:54:11.809106 9339 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
I20250904 07:54:11.812943 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:11.813140 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:11.813308 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 LEADER]: Raft consensus shutting down.
I20250904 07:54:11.813558 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:11.814237 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:11.814363 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 LEADER]: Raft consensus shutting down.
I20250904 07:54:11.814563 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Raft consensus is shut down!
W20250904 07:54:11.816246 8945 proxy.cc:239] Call had error, refreshing address and retrying: Service unavailable: reactor is shutting down (error 108) [suppressed 2 similar messages]
W20250904 07:54:11.822271 9277 connection.cc:441] server connection from 127.0.0.1:45628 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45628 (request call id 0) could send its response
I20250904 07:54:11.830098 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:54:11.833436 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:11.835655 9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:11.835821 9349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:11.836789 413 server_base.cc:1047] running on GCE node
W20250904 07:54:11.837704 9352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:11.838060 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:11.838129 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:11.838193 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972451838188 us; error 0 us; skew 500 ppm
I20250904 07:54:11.838330 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:11.839455 413 webserver.cc:480] Webserver started at http://127.0.103.67:39087/ using document root <none> and password file <none>
I20250904 07:54:11.839650 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:11.839704 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:11.841286 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250904 07:54:11.842490 9357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:11.842841 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:11.842937 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "47e41d546b88492da3428adbff442243"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:11.843029 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:11.849560 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:11.849928 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:11.850322 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:11.855379 413 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 07:54:11.855448 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250904 07:54:11.857607 9364 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:11.859247 413 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 07:54:11.859313 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250904 07:54:11.874526 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44843
I20250904 07:54:11.874549 9421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44843 every 8 connection(s)
I20250904 07:54:11.876839 9364 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=13 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:11.877367 9364 tablet_bootstrap.cc:492] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:11.877676 9364 ts_tablet_manager.cc:1397] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20250904 07:54:11.878295 9364 raft_consensus.cc:357] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.878487 9364 raft_consensus.cc:738] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:11.878731 9364 consensus_queue.cc:260] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 13.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:11.879398 9364 ts_tablet_manager.cc:1428] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:54:11.879655 9364 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap starting.
I20250904 07:54:11.882174 9422 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:11.882300 9422 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:11.882607 9422 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:11.883781 6067 ts_manager.cc:194] Re-registered known tserver with Master: 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:11.885130 6067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44152
I20250904 07:54:11.886135 9422 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
W20250904 07:54:11.890055 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.894299 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:11.903239 9364 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:11.903606 9364 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Bootstrap complete.
I20250904 07:54:11.903833 9364 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20250904 07:54:11.904340 9364 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.904486 9364 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41d546b88492da3428adbff442243, State: Initialized, Role: FOLLOWER
I20250904 07:54:11.904681 9364 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 15.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:11.905097 9364 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
W20250904 07:54:11.909114 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.919332 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.930979 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.943270 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.954620 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:11.968227 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.979499 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:11.993760 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.012372 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.026687 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.045068 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.062938 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.082230 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.102675 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.127518 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.152057 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.176534 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.200166 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.225561 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.252030 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.282979 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.313671 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.341257 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.373162 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.402706 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.436364 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.472527 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.505239 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.542915 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.579957 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.615720 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.653553 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.695850 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.734484 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.776429 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.818575 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.862345 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:12.909330 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
W20250904 07:54:12.956605 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:13.004503 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:13.050223 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:13.088328 9429 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:13.088498 9429 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.089313 9429 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935)
I20250904 07:54:13.096220 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:13.096221 9314 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "47e41d546b88492da3428adbff442243" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:13.096743 9358 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e41d546b88492da3428adbff442243; no voters: 6a2e703f66a34a598ab3d9646fa5db40, d888acb801154571b952c5dba8b792d7
I20250904 07:54:13.096963 9429 raft_consensus.cc:2747] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20250904 07:54:13.099149 9377 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54546: Illegal state: replica 47e41d546b88492da3428adbff442243 is not leader of this config: current role FOLLOWER
W20250904 07:54:13.148783 9178 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44790: Illegal state: replica 6a2e703f66a34a598ab3d9646fa5db40 is not leader of this config: current role FOLLOWER
W20250904 07:54:13.197824 9078 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49066: Illegal state: replica d67ae74f80964d33aaf83b8e54ed263c is not leader of this config: current role FOLLOWER
I20250904 07:54:13.210582 9434 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:13.210738 9434 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.211238 9434 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.212183 9435 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:13.212406 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:13.212374 9435 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.212630 9198 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 15.
I20250904 07:54:13.212971 9435 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 47e41d546b88492da3428adbff442243 (127.0.103.67:44843), d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:13.213256 9059 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters:
I20250904 07:54:13.213510 9434 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Leader pre-election won for term 16
I20250904 07:54:13.213582 9434 raft_consensus.cc:491] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:13.213668 9434 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 15 FOLLOWER]: Advancing to term 16
I20250904 07:54:13.214238 9098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "d67ae74f80964d33aaf83b8e54ed263c" is_pre_election: true
I20250904 07:54:13.216356 9434 raft_consensus.cc:513] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.216630 9098 raft_consensus.cc:2391] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a2e703f66a34a598ab3d9646fa5db40 in current term 16: Already voted for candidate d67ae74f80964d33aaf83b8e54ed263c in this term.
I20250904 07:54:13.217084 9434 leader_election.cc:290] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 16 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.217487 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:13.217626 9198 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 15 FOLLOWER]: Advancing to term 16
I20250904 07:54:13.218777 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:13.218855 9396 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "d67ae74f80964d33aaf83b8e54ed263c" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:13.218942 9397 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 15.
I20250904 07:54:13.219548 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d54fed14914b4487805737cebff4717a" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 16 candidate_status { last_received { term: 15 index: 32 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:13.219691 9397 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 15.
I20250904 07:54:13.219951 9162 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters: d67ae74f80964d33aaf83b8e54ed263c
I20250904 07:54:13.220252 9198 raft_consensus.cc:2466] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67ae74f80964d33aaf83b8e54ed263c in term 16.
I20250904 07:54:13.220394 9435 raft_consensus.cc:2762] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20250904 07:54:13.220654 9059 leader_election.cc:304] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a2e703f66a34a598ab3d9646fa5db40, d67ae74f80964d33aaf83b8e54ed263c; no voters: 47e41d546b88492da3428adbff442243
I20250904 07:54:13.220888 9434 raft_consensus.cc:2802] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Leader election won for term 16
I20250904 07:54:13.221190 9434 raft_consensus.cc:695] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 LEADER]: Becoming Leader. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Running, Role: LEADER
I20250904 07:54:13.221405 9434 consensus_queue.cc:237] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 15.32, Last appended by leader: 32, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.223377 6067 catalog_manager.cc:5582] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c reported cstate change: term changed from 15 to 16, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68). New cstate: current_term: 16 leader_uuid: "d67ae74f80964d33aaf83b8e54ed263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:13.252107 9397 raft_consensus.cc:3058] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Advancing to term 16
I20250904 07:54:13.252167 9198 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 16 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 15 index: 32. Preceding OpId from leader: term: 16 index: 34. (index mismatch)
I20250904 07:54:13.252609 9440 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250904 07:54:13.254822 9397 raft_consensus.cc:1273] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 16 FOLLOWER]: Refusing update from remote peer d67ae74f80964d33aaf83b8e54ed263c: Log matching property violated. Preceding OpId in replica: term: 15 index: 32. Preceding OpId from leader: term: 16 index: 34. (index mismatch)
I20250904 07:54:13.256322 9440 consensus_queue.cc:1035] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250904 07:54:13.258823 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:54:13.270205 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:13.270378 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.270573 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 LEADER]: Raft consensus shutting down.
I20250904 07:54:13.270803 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.293085 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:54:13.295926 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:13.298050 9454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.298179 9455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.298553 9457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.298779 413 server_base.cc:1047] running on GCE node
I20250904 07:54:13.298954 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:13.299006 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:13.299048 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972453299044 us; error 0 us; skew 500 ppm
I20250904 07:54:13.299173 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:13.300276 413 webserver.cc:480] Webserver started at http://127.0.103.68:39595/ using document root <none> and password file <none>
I20250904 07:54:13.300453 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:13.300506 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250904 07:54:13.302012 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.303045 9462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.303350 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.303442 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
uuid: "d67ae74f80964d33aaf83b8e54ed263c"
format_stamp: "Formatted at 2025-09-04 07:53:46 on dist-test-slave-ghlz"
I20250904 07:54:13.303534 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:13.304056 9435 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:13.304193 9435 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.305032 9435 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.305568 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:13.305740 9397 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 13.
I20250904 07:54:13.306128 9162 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:13.306321 9435 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20250904 07:54:13.306394 9435 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 47e41d546b88492da3428adbff442243)
I20250904 07:54:13.306480 9435 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:13.308332 9435 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.309100 9435 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 election: Requested vote from peers d888acb801154571b952c5dba8b792d7 (127.0.103.66:32779), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.309641 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:13.309788 9397 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:13.311841 9314 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7" is_pre_election: true
I20250904 07:54:13.311986 9397 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 14.
I20250904 07:54:13.312027 9314 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 13.
I20250904 07:54:13.312319 9313 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "6a2e703f66a34a598ab3d9646fa5db40" candidate_term: 14 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "d888acb801154571b952c5dba8b792d7"
I20250904 07:54:13.312510 9313 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 13 FOLLOWER]: Advancing to term 14
I20250904 07:54:13.315670 9313 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e703f66a34a598ab3d9646fa5db40 in term 14.
I20250904 07:54:13.316107 9162 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e41d546b88492da3428adbff442243, 6a2e703f66a34a598ab3d9646fa5db40; no voters:
I20250904 07:54:13.316489 9435 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Leader election won for term 14
I20250904 07:54:13.316824 9435 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 LEADER]: Becoming Leader. State: Replica: 6a2e703f66a34a598ab3d9646fa5db40, State: Running, Role: LEADER
I20250904 07:54:13.317109 9435 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 13.25, Last appended by leader: 25, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.318867 9467 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:13.318974 9467 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.319280 6067 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 reported cstate change: term changed from 13 to 14, leader changed from 47e41d546b88492da3428adbff442243 (127.0.103.67) to 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65). New cstate: current_term: 14 leader_uuid: "6a2e703f66a34a598ab3d9646fa5db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:13.324946 9467 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.331102 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 15 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40" is_pre_election: true
I20250904 07:54:13.331305 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:13.331763 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:13.332296 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:13.333444 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 15 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243" is_pre_election: true
I20250904 07:54:13.333528 9474 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 07:54:13.333585 9397 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 14.
I20250904 07:54:13.333894 9278 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:13.334132 9467 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20250904 07:54:13.334203 9467 raft_consensus.cc:491] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:13.334282 9467 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 14 FOLLOWER]: Advancing to term 15
I20250904 07:54:13.335891 413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 07:54:13.335956 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250904 07:54:13.335951 9467 raft_consensus.cc:513] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.336084 413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 07:54:13.336460 9467 leader_election.cc:290] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 15 election: Requested vote from peers 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65:38935), 47e41d546b88492da3428adbff442243 (127.0.103.67:44843)
I20250904 07:54:13.336768 9198 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 15 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "6a2e703f66a34a598ab3d9646fa5db40"
I20250904 07:54:13.336817 9397 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" candidate_uuid: "d888acb801154571b952c5dba8b792d7" candidate_term: 15 candidate_status { last_received { term: 13 index: 25 } } ignore_live_leader: false dest_uuid: "47e41d546b88492da3428adbff442243"
I20250904 07:54:13.336982 9397 raft_consensus.cc:3058] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 14 FOLLOWER]: Advancing to term 15
I20250904 07:54:13.338649 413 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 07:54:13.338712 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20250904 07:54:13.338757 9474 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap starting.
I20250904 07:54:13.339651 9397 raft_consensus.cc:2466] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d888acb801154571b952c5dba8b792d7 in term 15.
I20250904 07:54:13.340030 9278 leader_election.cc:304] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47e41d546b88492da3428adbff442243, d888acb801154571b952c5dba8b792d7; no voters: 6a2e703f66a34a598ab3d9646fa5db40
I20250904 07:54:13.340252 9467 raft_consensus.cc:2802] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 FOLLOWER]: Leader election won for term 15
I20250904 07:54:13.340610 9467 raft_consensus.cc:695] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 LEADER]: Becoming Leader. State: Replica: d888acb801154571b952c5dba8b792d7, State: Running, Role: LEADER
I20250904 07:54:13.340844 9467 consensus_queue.cc:237] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 13.25, Last appended by leader: 25, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } }
I20250904 07:54:13.343636 6067 catalog_manager.cc:5582] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 reported cstate change: term changed from 14 to 15, leader changed from 6a2e703f66a34a598ab3d9646fa5db40 (127.0.103.65) to d888acb801154571b952c5dba8b792d7 (127.0.103.66). New cstate: current_term: 15 leader_uuid: "d888acb801154571b952c5dba8b792d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d888acb801154571b952c5dba8b792d7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 32779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:13.363932 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42085
I20250904 07:54:13.363948 9533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42085 every 8 connection(s)
I20250904 07:54:13.369836 9474 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=19 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 07:54:13.370398 9474 tablet_bootstrap.cc:492] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Bootstrap complete.
I20250904 07:54:13.370760 9474 ts_tablet_manager.cc:1397] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.015s
I20250904 07:54:13.371718 9474 raft_consensus.cc:357] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.371984 9474 raft_consensus.cc:738] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67ae74f80964d33aaf83b8e54ed263c, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.372277 9474 consensus_queue.cc:260] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 16.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e703f66a34a598ab3d9646fa5db40" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38935 } } peers { permanent_uuid: "47e41d546b88492da3428adbff442243" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44843 } } peers { permanent_uuid: "d67ae74f80964d33aaf83b8e54ed263c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42085 } }
I20250904 07:54:13.382522 9474 ts_tablet_manager.cc:1428] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20250904 07:54:13.386298 9534 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39793
I20250904 07:54:13.386408 9534 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:13.386734 9534 heartbeater.cc:507] Master 127.0.103.126:39793 requested a full tablet report, sending...
I20250904 07:54:13.387694 6067 ts_manager.cc:194] Re-registered known tserver with Master: d67ae74f80964d33aaf83b8e54ed263c (127.0.103.68:42085)
I20250904 07:54:13.388659 6067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44160
I20250904 07:54:13.389719 9534 heartbeater.cc:499] Master 127.0.103.126:39793 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:9999: Failure
Expected equality of these values:
kNumRows
Which is: 32
row_count
Which is: 31
Google Test trace:
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/client-test.cc:9995: read mode READ_LATEST
I20250904 07:54:13.395499 413 tablet_server.cc:178] TabletServer@127.0.103.65:38935 shutting down...
I20250904 07:54:13.403126 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:13.403373 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.403620 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 16 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:13.403851 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 6a2e703f66a34a598ab3d9646fa5db40 [term 16 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.404562 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.404656 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 LEADER]: Raft consensus shutting down.
I20250904 07:54:13.404835 413 pending_rounds.cc:70] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Trying to abort 1 pending ops.
I20250904 07:54:13.404927 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40: Aborting op as it isn't in flight: id { term: 14 index: 26 } timestamp: 7196559168788971520 op_type: NO_OP noop_request { }
I20250904 07:54:13.405040 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:13.405107 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 6a2e703f66a34a598ab3d9646fa5db40 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.409315 413 tablet_server.cc:195] TabletServer@127.0.103.65:38935 shutdown complete.
I20250904 07:54:13.412184 413 tablet_server.cc:178] TabletServer@127.0.103.66:32779 shutting down...
I20250904 07:54:13.420459 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:13.420642 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.420783 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 LEADER]: Raft consensus shutting down.
I20250904 07:54:13.420992 413 pending_rounds.cc:70] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Trying to abort 2 pending ops.
I20250904 07:54:13.421034 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Aborting op as it isn't in flight: id { term: 15 index: 26 } timestamp: 7196559168886607872 op_type: NO_OP noop_request { }
I20250904 07:54:13.421120 413 raft_consensus.cc:2887] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:13.421182 413 pending_rounds.cc:77] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7: Aborting op as it isn't in flight: id { term: 13 index: 25 } timestamp: 7196559162561720320 op_type: WRITE_OP write_request { tablet_id: "abd498e731794d59bc5fcf9a0de7a6d3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" write_default_value: "90\000\000" immutable: false } } row_operations { rows: "\001\017\000\036\000\000\000<\000\000\000\000\000\000\000\000\000\000\000\010\000\000\000\000\000\000\000Z\000\000\000""\001\017\000\036\000\000\000<\000\000\000\000\000\000\000\000\000\000\000\010\000\000\000\000\000\000\000Z\000\000\000" indirect_data: "hello 30""hello 30" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7196559162161258496 authz_token { token_data: "\010\267\215\345\305\006\"3\n\005slave\022*\n e7ec7b5e119243009ba67e3375146cff\020\001\030\001 \001(\001" signature: "+\221\371\231E\355\'\223z9\'\374\242;\\\345/\310O\357\262\202lnv\355\366\343\303\377\261]j\235\256W3\247\005^@\343\274FMzU\343\026\274\265\277\244\300?\242A\264\"\\\275Y\347\254""+\221\371\231E\355\'\223z9\'\374\242;\\\345/\310O\357\262\202lnv\355\366\343\303\377\261]j\235\256W3\247\005^@\343\274FMzU\343\026\274\265\277\244\300?\242A\264\"\\\275Y\347\254" signing_key_seq_num: 0 } } request_id { client_id: "8f2ade9ba4334c76894179e301c67080" seq_no: 30 first_incomplete_seq_no: 30 attempt_no: 0 }
I20250904 07:54:13.421563 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P d888acb801154571b952c5dba8b792d7 [term 15 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.435160 413 tablet_server.cc:195] TabletServer@127.0.103.66:32779 shutdown complete.
I20250904 07:54:13.437767 413 tablet_server.cc:178] TabletServer@127.0.103.67:44843 shutting down...
I20250904 07:54:13.444999 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:13.445218 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.445382 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 16 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:13.445538 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P 47e41d546b88492da3428adbff442243 [term 16 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.446223 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.446319 413 raft_consensus.cc:2241] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:13.446413 413 raft_consensus.cc:2270] T abd498e731794d59bc5fcf9a0de7a6d3 P 47e41d546b88492da3428adbff442243 [term 15 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.464531 413 tablet_server.cc:195] TabletServer@127.0.103.67:44843 shutdown complete.
I20250904 07:54:13.467409 413 tablet_server.cc:178] TabletServer@127.0.103.68:42085 shutting down...
I20250904 07:54:13.474623 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:13.474900 413 tablet_replica.cc:331] stopping tablet replica
I20250904 07:54:13.475070 413 raft_consensus.cc:2241] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:13.475212 413 raft_consensus.cc:2270] T d54fed14914b4487805737cebff4717a P d67ae74f80964d33aaf83b8e54ed263c [term 16 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.488441 413 tablet_server.cc:195] TabletServer@127.0.103.68:42085 shutdown complete.
I20250904 07:54:13.491174 413 master.cc:561] Master@127.0.103.126:39793 shutting down...
I20250904 07:54:13.498409 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:13.498608 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:13.498701 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 26ea8ca3f3ef4e73955b5e5408a9bca7: stopping tablet replica
W20250904 07:54:13.511571 413 rolling_log.cc:182] Unable to compress old log file: Not found: Unable to open input file to compress: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0/client-test.dist-test-slave-ghlz.slave.diagnostics.20250904-075346.0.413: No such file or directory (error 2)
I20250904 07:54:13.511828 413 master.cc:583] Master@127.0.103.126:39793 shutdown complete.
I20250904 07:54:13.517241 413 test_util.cc:183] -----------------------------------------------
I20250904 07:54:13.517314 413 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers.1756972373064810-413-0
[ FAILED ] MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers (27077 ms)
[----------] 1 test from MultiTServerClientTest (27077 ms total)
[----------] 1 test from ClientTestAutoIncrementingColumn
[ RUN ] ClientTestAutoIncrementingColumn.AlterOperationNegatives
I20250904 07:54:13.519210 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:38551
I20250904 07:54:13.519678 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:13.521508 9541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.524696 9542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.524755 413 server_base.cc:1047] running on GCE node
W20250904 07:54:13.524751 9544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.525164 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:13.525228 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:13.525267 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972453525255 us; error 0 us; skew 500 ppm
I20250904 07:54:13.525389 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:13.526430 413 webserver.cc:480] Webserver started at http://127.0.103.126:43769/ using document root <none> and password file <none>
I20250904 07:54:13.526620 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:13.526695 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:13.526793 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:13.527319 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "c7102e03366748ea801b1323204a3e4e"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.529676 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:54:13.531011 9549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.531342 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.531433 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "c7102e03366748ea801b1323204a3e4e"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.531580 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:13.584229 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:13.584719 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:13.594748 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:38551
I20250904 07:54:13.594779 9600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:38551 every 8 connection(s)
I20250904 07:54:13.595917 9601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.600050 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e: Bootstrap starting.
I20250904 07:54:13.601161 9601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.602480 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e: No bootstrap required, opened a new log
I20250904 07:54:13.603063 9601 raft_consensus.cc:357] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER }
I20250904 07:54:13.603193 9601 raft_consensus.cc:383] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.603272 9601 raft_consensus.cc:738] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7102e03366748ea801b1323204a3e4e, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.603480 9601 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER }
I20250904 07:54:13.603600 9601 raft_consensus.cc:397] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:13.603681 9601 raft_consensus.cc:491] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:13.603765 9601 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:13.605573 9601 raft_consensus.cc:513] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER }
I20250904 07:54:13.605762 9601 leader_election.cc:304] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7102e03366748ea801b1323204a3e4e; no voters:
I20250904 07:54:13.606139 9601 leader_election.cc:290] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:13.606225 9604 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:13.606689 9604 raft_consensus.cc:695] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 1 LEADER]: Becoming Leader. State: Replica: c7102e03366748ea801b1323204a3e4e, State: Running, Role: LEADER
I20250904 07:54:13.606910 9604 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER }
I20250904 07:54:13.606968 9601 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:13.607573 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7102e03366748ea801b1323204a3e4e. Latest consensus state: current_term: 1 leader_uuid: "c7102e03366748ea801b1323204a3e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER } }
I20250904 07:54:13.607758 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:13.607879 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7102e03366748ea801b1323204a3e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7102e03366748ea801b1323204a3e4e" member_type: VOTER } }
I20250904 07:54:13.608043 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:13.613384 9609 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:13.614440 9609 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:13.615844 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:13.617017 9609 catalog_manager.cc:1349] Generated new cluster ID: 0451eb643e90462dabd1401083fbcd83
I20250904 07:54:13.617118 9609 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:13.635599 9609 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:13.636287 9609 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:13.647486 9609 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e: Generated new TSK 0
I20250904 07:54:13.647699 9609 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:13.680635 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:13.683099 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.683133 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.683772 413 server_base.cc:1047] running on GCE node
W20250904 07:54:13.684085 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.684371 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:13.684450 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:13.684520 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972453684499 us; error 0 us; skew 500 ppm
I20250904 07:54:13.684698 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:13.686025 413 webserver.cc:480] Webserver started at http://127.0.103.65:35821/ using document root <none> and password file <none>
I20250904 07:54:13.686239 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:13.686321 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:13.686465 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:13.687072 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "933dce0bfae14fd59e06e186927f86f7"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.689438 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:54:13.690851 9630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.691190 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.691304 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "933dce0bfae14fd59e06e186927f86f7"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.691422 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:13.717312 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:13.717749 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:13.718238 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:13.719026 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:13.719095 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.719195 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:13.719252 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.729766 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:37345
I20250904 07:54:13.729794 9692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:37345 every 8 connection(s)
I20250904 07:54:13.731588 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:13.734241 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.737069 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.738440 413 server_base.cc:1047] running on GCE node
W20250904 07:54:13.738629 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.738801 9693 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38551
I20250904 07:54:13.738948 9693 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:13.738986 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:13.739084 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:13.739142 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972453739121 us; error 0 us; skew 500 ppm
I20250904 07:54:13.739296 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:13.739310 9693 heartbeater.cc:507] Master 127.0.103.126:38551 requested a full tablet report, sending...
I20250904 07:54:13.740216 9566 ts_manager.cc:194] Registered new tserver with Master: 933dce0bfae14fd59e06e186927f86f7 (127.0.103.65:37345)
I20250904 07:54:13.740514 413 webserver.cc:480] Webserver started at http://127.0.103.66:41563/ using document root <none> and password file <none>
I20250904 07:54:13.740742 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:13.740831 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:13.740993 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:13.741667 9566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55546
I20250904 07:54:13.741686 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-1-root/instance:
uuid: "ad97ca80ca74471d8ae65b5746791660"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.744195 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250904 07:54:13.745551 9705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.745874 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.745975 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "ad97ca80ca74471d8ae65b5746791660"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.746083 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:13.771109 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:13.771510 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:13.771981 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:13.772713 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:13.772812 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.772935 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:13.772993 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.782960 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36461
I20250904 07:54:13.782990 9767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36461 every 8 connection(s)
I20250904 07:54:13.784430 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:13.786612 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.786885 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:13.787308 9774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:13.787868 413 server_base.cc:1047] running on GCE node
I20250904 07:54:13.788075 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:13.788142 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:13.788209 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972453788185 us; error 0 us; skew 500 ppm
I20250904 07:54:13.788372 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:13.789579 413 webserver.cc:480] Webserver started at http://127.0.103.67:44369/ using document root <none> and password file <none>
I20250904 07:54:13.789817 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:13.789923 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:13.790068 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:13.790488 9768 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38551
I20250904 07:54:13.790617 9768 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:13.790804 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-2-root/instance:
uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.790940 9768 heartbeater.cc:507] Master 127.0.103.126:38551 requested a full tablet report, sending...
I20250904 07:54:13.791836 9566 ts_manager.cc:194] Registered new tserver with Master: ad97ca80ca74471d8ae65b5746791660 (127.0.103.66:36461)
I20250904 07:54:13.792841 9566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55556
I20250904 07:54:13.793630 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250904 07:54:13.794971 9779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.795348 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:13.795444 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42"
format_stamp: "Formatted at 2025-09-04 07:54:13 on dist-test-slave-ghlz"
I20250904 07:54:13.795542 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ClientTestAutoIncrementingColumn.AlterOperationNegatives.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:13.832782 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:13.833225 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:13.833669 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 07:54:13.834357 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:13.834411 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.834478 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:13.834508 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.844985 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35015
I20250904 07:54:13.845012 9841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35015 every 8 connection(s)
I20250904 07:54:13.850858 9842 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38551
I20250904 07:54:13.850970 9842 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:13.851256 9842 heartbeater.cc:507] Master 127.0.103.126:38551 requested a full tablet report, sending...
I20250904 07:54:13.852013 9566 ts_manager.cc:194] Registered new tserver with Master: 5dc0c5fc8b2543799ea7af7c1c24dc42 (127.0.103.67:35015)
I20250904 07:54:13.852569 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006708175s
I20250904 07:54:13.852720 9566 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55568
I20250904 07:54:13.862457 9566 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55580:
name: "alter_operation_negatives_auto_incrementing_column"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "auto_incrementing_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
is_auto_incrementing: true
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250904 07:54:13.863027 9566 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table alter_operation_negatives_auto_incrementing_column in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:54:13.874861 9806 tablet_service.cc:1468] Processing CreateTablet for tablet d3ac8541c6114422a7e0a23b20eb44e0 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.874861 9807 tablet_service.cc:1468] Processing CreateTablet for tablet aa3c9f159af24e35868ccb7afe266379 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.875349 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ac8541c6114422a7e0a23b20eb44e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.875495 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c9f159af24e35868ccb7afe266379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.883857 9856 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Bootstrap starting.
I20250904 07:54:13.884974 9856 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.885496 9856 tablet_bootstrap.cc:654] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.887200 9856 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: No bootstrap required, opened a new log
I20250904 07:54:13.887338 9856 ts_tablet_manager.cc:1397] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250904 07:54:13.888507 9733 tablet_service.cc:1468] Processing CreateTablet for tablet aa3c9f159af24e35868ccb7afe266379 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.888717 9658 tablet_service.cc:1468] Processing CreateTablet for tablet aa3c9f159af24e35868ccb7afe266379 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.888926 9733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c9f159af24e35868ccb7afe266379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.888835 9657 tablet_service.cc:1468] Processing CreateTablet for tablet d3ac8541c6114422a7e0a23b20eb44e0 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.889091 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3c9f159af24e35868ccb7afe266379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.889318 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ac8541c6114422a7e0a23b20eb44e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.893154 9732 tablet_service.cc:1468] Processing CreateTablet for tablet d3ac8541c6114422a7e0a23b20eb44e0 (DEFAULT_TABLE table=alter_operation_negatives_auto_incrementing_column [id=0466966f92a94f14a659fa4d21eb0414]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:13.893452 9732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ac8541c6114422a7e0a23b20eb44e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:13.895624 9858 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Bootstrap starting.
I20250904 07:54:13.887899 9856 raft_consensus.cc:357] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.897156 9858 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.897892 9858 tablet_bootstrap.cc:654] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.897284 9856 raft_consensus.cc:383] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.898850 9856 raft_consensus.cc:738] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc0c5fc8b2543799ea7af7c1c24dc42, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.899240 9856 consensus_queue.cc:260] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.900010 9842 heartbeater.cc:499] Master 127.0.103.126:38551 was elected leader, sending a full tablet report...
I20250904 07:54:13.900137 9856 ts_tablet_manager.cc:1428] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250904 07:54:13.900920 9858 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: No bootstrap required, opened a new log
I20250904 07:54:13.901084 9856 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Bootstrap starting.
I20250904 07:54:13.901154 9858 ts_tablet_manager.cc:1397] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250904 07:54:13.902122 9858 raft_consensus.cc:357] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.902333 9858 raft_consensus.cc:383] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.902406 9858 raft_consensus.cc:738] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933dce0bfae14fd59e06e186927f86f7, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.902679 9856 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.902761 9858 consensus_queue.cc:260] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.903194 9856 tablet_bootstrap.cc:654] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.903640 9859 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: Bootstrap starting.
I20250904 07:54:13.905633 9859 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.906199 9859 tablet_bootstrap.cc:654] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.908082 9856 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: No bootstrap required, opened a new log
I20250904 07:54:13.908202 9856 ts_tablet_manager.cc:1397] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20250904 07:54:13.908437 9859 tablet_bootstrap.cc:492] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: No bootstrap required, opened a new log
I20250904 07:54:13.908593 9859 ts_tablet_manager.cc:1397] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:54:13.908716 9856 raft_consensus.cc:357] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.908934 9856 raft_consensus.cc:383] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.909039 9856 raft_consensus.cc:738] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc0c5fc8b2543799ea7af7c1c24dc42, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.909170 9693 heartbeater.cc:499] Master 127.0.103.126:38551 was elected leader, sending a full tablet report...
I20250904 07:54:13.909271 9856 consensus_queue.cc:260] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.909472 9859 raft_consensus.cc:357] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.909597 9859 raft_consensus.cc:383] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.909643 9859 raft_consensus.cc:738] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad97ca80ca74471d8ae65b5746791660, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.909699 9856 ts_tablet_manager.cc:1428] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:13.909852 9859 consensus_queue.cc:260] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:13.910537 9859 ts_tablet_manager.cc:1428] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:13.910514 9768 heartbeater.cc:499] Master 127.0.103.126:38551 was elected leader, sending a full tablet report...
I20250904 07:54:13.910800 9859 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: Bootstrap starting.
I20250904 07:54:13.911295 9858 ts_tablet_manager.cc:1428] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20250904 07:54:13.911796 9859 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.911866 9858 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Bootstrap starting.
I20250904 07:54:13.912220 9859 tablet_bootstrap.cc:654] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.912812 9858 tablet.cc:502] Time spent fetching auto increment counter: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:13.913684 9858 tablet_bootstrap.cc:654] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:13.917809 9858 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: No bootstrap required, opened a new log
I20250904 07:54:13.917936 9858 ts_tablet_manager.cc:1397] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250904 07:54:13.918166 9859 tablet_bootstrap.cc:492] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: No bootstrap required, opened a new log
I20250904 07:54:13.918267 9859 ts_tablet_manager.cc:1397] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20250904 07:54:13.918500 9858 raft_consensus.cc:357] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.918622 9858 raft_consensus.cc:383] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.918684 9858 raft_consensus.cc:738] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933dce0bfae14fd59e06e186927f86f7, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.918763 9859 raft_consensus.cc:357] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.918892 9859 raft_consensus.cc:383] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:13.918958 9859 raft_consensus.cc:738] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad97ca80ca74471d8ae65b5746791660, State: Initialized, Role: FOLLOWER
I20250904 07:54:13.918872 9858 consensus_queue.cc:260] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.919145 9859 consensus_queue.cc:260] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:13.919449 9858 ts_tablet_manager.cc:1428] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:13.919499 9859 ts_tablet_manager.cc:1428] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250904 07:54:13.980912 9694 tablet.cc:2378] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 07:54:14.157368 9862 raft_consensus.cc:491] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:14.157557 9862 raft_consensus.cc:513] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:14.158234 9862 leader_election.cc:290] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc0c5fc8b2543799ea7af7c1c24dc42 (127.0.103.67:35015), ad97ca80ca74471d8ae65b5746791660 (127.0.103.66:36461)
I20250904 07:54:14.163182 9817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3c9f159af24e35868ccb7afe266379" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" is_pre_election: true
I20250904 07:54:14.163448 9817 raft_consensus.cc:2466] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 0.
I20250904 07:54:14.163692 9743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3c9f159af24e35868ccb7afe266379" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad97ca80ca74471d8ae65b5746791660" is_pre_election: true
I20250904 07:54:14.163892 9743 raft_consensus.cc:2466] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 0.
I20250904 07:54:14.163888 9633 leader_election.cc:304] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc0c5fc8b2543799ea7af7c1c24dc42, 933dce0bfae14fd59e06e186927f86f7; no voters:
I20250904 07:54:14.164213 9862 raft_consensus.cc:2802] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:54:14.164306 9862 raft_consensus.cc:491] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:14.164384 9862 raft_consensus.cc:3058] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.166357 9862 raft_consensus.cc:513] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:14.166846 9862 leader_election.cc:290] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc0c5fc8b2543799ea7af7c1c24dc42 (127.0.103.67:35015), ad97ca80ca74471d8ae65b5746791660 (127.0.103.66:36461)
I20250904 07:54:14.167191 9817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3c9f159af24e35868ccb7afe266379" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42"
I20250904 07:54:14.167202 9743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3c9f159af24e35868ccb7afe266379" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad97ca80ca74471d8ae65b5746791660"
I20250904 07:54:14.167315 9817 raft_consensus.cc:3058] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.167357 9743 raft_consensus.cc:3058] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.169234 9817 raft_consensus.cc:2466] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 1.
I20250904 07:54:14.169235 9743 raft_consensus.cc:2466] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 1.
I20250904 07:54:14.169574 9633 leader_election.cc:304] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc0c5fc8b2543799ea7af7c1c24dc42, 933dce0bfae14fd59e06e186927f86f7; no voters:
I20250904 07:54:14.169800 9862 raft_consensus.cc:2802] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.170073 9862 raft_consensus.cc:695] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: Becoming Leader. State: Replica: 933dce0bfae14fd59e06e186927f86f7, State: Running, Role: LEADER
I20250904 07:54:14.170276 9862 consensus_queue.cc:237] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } }
I20250904 07:54:14.172267 9564 catalog_manager.cc:5582] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 933dce0bfae14fd59e06e186927f86f7 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "933dce0bfae14fd59e06e186927f86f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:14.205132 9862 raft_consensus.cc:491] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:14.205302 9862 raft_consensus.cc:513] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:14.205884 9862 leader_election.cc:290] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad97ca80ca74471d8ae65b5746791660 (127.0.103.66:36461), 5dc0c5fc8b2543799ea7af7c1c24dc42 (127.0.103.67:35015)
I20250904 07:54:14.206262 9743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3ac8541c6114422a7e0a23b20eb44e0" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad97ca80ca74471d8ae65b5746791660" is_pre_election: true
I20250904 07:54:14.206283 9817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3ac8541c6114422a7e0a23b20eb44e0" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" is_pre_election: true
I20250904 07:54:14.206411 9743 raft_consensus.cc:2466] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 0.
I20250904 07:54:14.206439 9817 raft_consensus.cc:2466] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 0.
I20250904 07:54:14.206768 9633 leader_election.cc:304] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5dc0c5fc8b2543799ea7af7c1c24dc42, 933dce0bfae14fd59e06e186927f86f7, ad97ca80ca74471d8ae65b5746791660; no voters:
I20250904 07:54:14.206981 9862 raft_consensus.cc:2802] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:54:14.207048 9862 raft_consensus.cc:491] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:14.207089 9862 raft_consensus.cc:3058] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.209115 9862 raft_consensus.cc:513] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:14.209565 9862 leader_election.cc:290] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 election: Requested vote from peers ad97ca80ca74471d8ae65b5746791660 (127.0.103.66:36461), 5dc0c5fc8b2543799ea7af7c1c24dc42 (127.0.103.67:35015)
I20250904 07:54:14.209933 9817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3ac8541c6114422a7e0a23b20eb44e0" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42"
I20250904 07:54:14.209933 9743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3ac8541c6114422a7e0a23b20eb44e0" candidate_uuid: "933dce0bfae14fd59e06e186927f86f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad97ca80ca74471d8ae65b5746791660"
I20250904 07:54:14.210115 9817 raft_consensus.cc:3058] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.210132 9743 raft_consensus.cc:3058] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.212735 9743 raft_consensus.cc:2466] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 1.
I20250904 07:54:14.212735 9817 raft_consensus.cc:2466] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933dce0bfae14fd59e06e186927f86f7 in term 1.
I20250904 07:54:14.213155 9633 leader_election.cc:304] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc0c5fc8b2543799ea7af7c1c24dc42, 933dce0bfae14fd59e06e186927f86f7; no voters:
I20250904 07:54:14.213351 9862 raft_consensus.cc:2802] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.213480 9862 raft_consensus.cc:695] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: Becoming Leader. State: Replica: 933dce0bfae14fd59e06e186927f86f7, State: Running, Role: LEADER
I20250904 07:54:14.213716 9862 consensus_queue.cc:237] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } }
I20250904 07:54:14.215950 9564 catalog_manager.cc:5582] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 933dce0bfae14fd59e06e186927f86f7 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "933dce0bfae14fd59e06e186927f86f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933dce0bfae14fd59e06e186927f86f7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad97ca80ca74471d8ae65b5746791660" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dc0c5fc8b2543799ea7af7c1c24dc42" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35015 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:14.234648 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:14.242102 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:14.242414 413 tablet_replica.cc:331] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: stopping tablet replica
I20250904 07:54:14.242584 413 raft_consensus.cc:2241] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.242767 413 pending_rounds.cc:70] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Trying to abort 1 pending ops.
I20250904 07:54:14.242797 413 pending_rounds.cc:77] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196559172461453312 op_type: NO_OP noop_request { }
I20250904 07:54:14.242880 413 raft_consensus.cc:2887] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:14.242933 413 raft_consensus.cc:2270] T d3ac8541c6114422a7e0a23b20eb44e0 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.243736 413 tablet_replica.cc:331] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: stopping tablet replica
I20250904 07:54:14.243860 413 raft_consensus.cc:2241] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.244040 413 pending_rounds.cc:70] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Trying to abort 1 pending ops.
I20250904 07:54:14.244079 413 pending_rounds.cc:77] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196559172283518976 op_type: NO_OP noop_request { }
I20250904 07:54:14.244163 413 raft_consensus.cc:2887] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250904 07:54:14.244246 413 raft_consensus.cc:2270] T aa3c9f159af24e35868ccb7afe266379 P 933dce0bfae14fd59e06e186927f86f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.258332 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:14.261377 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250904 07:54:14.268566 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:14.268848 413 tablet_replica.cc:331] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660: stopping tablet replica
I20250904 07:54:14.269034 413 raft_consensus.cc:2241] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:14.269182 413 raft_consensus.cc:2270] T d3ac8541c6114422a7e0a23b20eb44e0 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.269743 413 tablet_replica.cc:331] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660: stopping tablet replica
I20250904 07:54:14.269866 413 raft_consensus.cc:2241] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:14.269980 413 raft_consensus.cc:2270] T aa3c9f159af24e35868ccb7afe266379 P ad97ca80ca74471d8ae65b5746791660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.283208 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250904 07:54:14.285955 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250904 07:54:14.294081 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:14.294353 413 tablet_replica.cc:331] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42: stopping tablet replica
I20250904 07:54:14.294502 413 raft_consensus.cc:2241] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:14.294632 413 raft_consensus.cc:2270] T aa3c9f159af24e35868ccb7afe266379 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.295181 413 tablet_replica.cc:331] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42: stopping tablet replica
I20250904 07:54:14.295315 413 raft_consensus.cc:2241] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:14.295450 413 raft_consensus.cc:2270] T d3ac8541c6114422a7e0a23b20eb44e0 P 5dc0c5fc8b2543799ea7af7c1c24dc42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.308931 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250904 07:54:14.311635 413 master.cc:561] Master@127.0.103.126:38551 shutting down...
I20250904 07:54:14.318514 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.318693 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.318794 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P c7102e03366748ea801b1323204a3e4e: stopping tablet replica
I20250904 07:54:14.332149 413 master.cc:583] Master@127.0.103.126:38551 shutdown complete.
[ OK ] ClientTestAutoIncrementingColumn.AlterOperationNegatives (830 ms)
[----------] 1 test from ClientTestAutoIncrementingColumn (830 ms total)
[----------] 1 test from KeepAlivePeriodically/KeepAlivePeriodicallyTest
[ RUN ] KeepAlivePeriodically/KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers/1
I20250904 07:54:14.350217 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37359
I20250904 07:54:14.350713 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:14.353421 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.353797 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.353808 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.354207 413 server_base.cc:1047] running on GCE node
I20250904 07:54:14.354414 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.354472 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.354517 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454354513 us; error 0 us; skew 500 ppm
I20250904 07:54:14.354663 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.355712 413 webserver.cc:480] Webserver started at http://127.0.103.126:43295/ using document root <none> and password file <none>
I20250904 07:54:14.355902 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.355963 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.356081 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.356659 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "69c307c831f64e00b0debc8e2f1c9873"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.358920 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250904 07:54:14.360257 9880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.360723 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:14.360816 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "69c307c831f64e00b0debc8e2f1c9873"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.360951 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.398409 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.398869 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.409662 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37359
I20250904 07:54:14.409679 9941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:37359 every 8 connection(s)
I20250904 07:54:14.410785 9942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.414916 9942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873: Bootstrap starting.
I20250904 07:54:14.415963 9942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.417244 9942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873: No bootstrap required, opened a new log
I20250904 07:54:14.417764 9942 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER }
I20250904 07:54:14.417874 9942 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.417913 9942 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c307c831f64e00b0debc8e2f1c9873, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.418076 9942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER }
I20250904 07:54:14.418186 9942 raft_consensus.cc:397] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:14.418228 9942 raft_consensus.cc:491] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:14.418278 9942 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.419973 9942 raft_consensus.cc:513] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER }
I20250904 07:54:14.420120 9942 leader_election.cc:304] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c307c831f64e00b0debc8e2f1c9873; no voters:
I20250904 07:54:14.420425 9942 leader_election.cc:290] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:14.420540 9945 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.420953 9945 raft_consensus.cc:695] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 1 LEADER]: Becoming Leader. State: Replica: 69c307c831f64e00b0debc8e2f1c9873, State: Running, Role: LEADER
I20250904 07:54:14.421144 9945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER }
I20250904 07:54:14.421217 9942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:14.421816 9946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c307c831f64e00b0debc8e2f1c9873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER } }
I20250904 07:54:14.421881 9947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c307c831f64e00b0debc8e2f1c9873. Latest consensus state: current_term: 1 leader_uuid: "69c307c831f64e00b0debc8e2f1c9873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c307c831f64e00b0debc8e2f1c9873" member_type: VOTER } }
I20250904 07:54:14.422039 9946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:14.422056 9947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:14.423238 9950 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:14.424280 9950 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:14.425885 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:14.426771 9950 catalog_manager.cc:1349] Generated new cluster ID: c23482cf9fcb45838b10140c250f7046
I20250904 07:54:14.426842 9950 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:14.464926 9950 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:14.465643 9950 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:14.471781 9950 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873: Generated new TSK 0
I20250904 07:54:14.471958 9950 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:14.490481 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:14.492925 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.493311 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.493309 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.493701 413 server_base.cc:1047] running on GCE node
I20250904 07:54:14.493896 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.493947 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.493984 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454493981 us; error 0 us; skew 500 ppm
I20250904 07:54:14.494132 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.495150 413 webserver.cc:480] Webserver started at http://127.0.103.65:37013/ using document root <none> and password file <none>
I20250904 07:54:14.495343 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.495401 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.495522 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.496081 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "c0d8962738a24dff98cecf26cfc52d75"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.498407 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:54:14.499753 9971 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.500036 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:14.500135 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "c0d8962738a24dff98cecf26cfc52d75"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.500231 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.535851 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.536288 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.537883 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:14.538019 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.538156 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:14.538252 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.555552 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45233
I20250904 07:54:14.555573 10041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45233 every 8 connection(s)
I20250904 07:54:14.556926 10042 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37359
I20250904 07:54:14.557029 10042 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:14.557260 10042 heartbeater.cc:507] Master 127.0.103.126:37359 requested a full tablet report, sending...
I20250904 07:54:14.557932 9897 ts_manager.cc:194] Registered new tserver with Master: c0d8962738a24dff98cecf26cfc52d75 (127.0.103.65:45233)
I20250904 07:54:14.558683 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002253597s
I20250904 07:54:14.558883 9897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53116
I20250904 07:54:14.568634 9897 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53124:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:54:14.570297 10042 heartbeater.cc:499] Master 127.0.103.126:37359 was elected leader, sending a full tablet report...
I20250904 07:54:14.580217 10002 tablet_service.cc:1468] Processing CreateTablet for tablet 3f3cf82ec55147f0bb55b7bf4688b021 (DEFAULT_TABLE table=client-testtb [id=6b59e3a8c7b1463fab235da8903444a4]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:54:14.580276 10001 tablet_service.cc:1468] Processing CreateTablet for tablet 282d67aef15e428483cf403855c44e5c (DEFAULT_TABLE table=client-testtb [id=6b59e3a8c7b1463fab235da8903444a4]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:54:14.580647 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f3cf82ec55147f0bb55b7bf4688b021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.580849 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282d67aef15e428483cf403855c44e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.585927 10052 tablet_bootstrap.cc:492] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: Bootstrap starting.
I20250904 07:54:14.586961 10052 tablet_bootstrap.cc:654] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.588418 10052 tablet_bootstrap.cc:492] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: No bootstrap required, opened a new log
I20250904 07:54:14.588519 10052 ts_tablet_manager.cc:1397] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:14.589035 10052 raft_consensus.cc:357] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.589154 10052 raft_consensus.cc:383] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.589191 10052 raft_consensus.cc:738] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d8962738a24dff98cecf26cfc52d75, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.589344 10052 consensus_queue.cc:260] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.589447 10052 raft_consensus.cc:397] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:14.589488 10052 raft_consensus.cc:491] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:14.589537 10052 raft_consensus.cc:3058] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.591219 10052 raft_consensus.cc:513] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.591375 10052 leader_election.cc:304] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d8962738a24dff98cecf26cfc52d75; no voters:
I20250904 07:54:14.591709 10052 leader_election.cc:290] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:14.591832 10054 raft_consensus.cc:2802] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.592237 10054 raft_consensus.cc:695] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 1 LEADER]: Becoming Leader. State: Replica: c0d8962738a24dff98cecf26cfc52d75, State: Running, Role: LEADER
I20250904 07:54:14.592289 10052 ts_tablet_manager.cc:1428] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 07:54:14.592520 10054 consensus_queue.cc:237] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.592963 10052 tablet_bootstrap.cc:492] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: Bootstrap starting.
I20250904 07:54:14.594415 10052 tablet_bootstrap.cc:654] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.594414 9897 catalog_manager.cc:5582] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0d8962738a24dff98cecf26cfc52d75 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "c0d8962738a24dff98cecf26cfc52d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:14.595929 10052 tablet_bootstrap.cc:492] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: No bootstrap required, opened a new log
I20250904 07:54:14.596043 10052 ts_tablet_manager.cc:1397] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:14.596575 10052 raft_consensus.cc:357] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.596711 10052 raft_consensus.cc:383] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.596752 10052 raft_consensus.cc:738] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d8962738a24dff98cecf26cfc52d75, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.596946 10052 consensus_queue.cc:260] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.597051 10052 raft_consensus.cc:397] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:14.597117 10052 raft_consensus.cc:491] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:14.597198 10052 raft_consensus.cc:3058] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.599200 10052 raft_consensus.cc:513] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.599365 10052 leader_election.cc:304] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d8962738a24dff98cecf26cfc52d75; no voters:
I20250904 07:54:14.599506 10052 leader_election.cc:290] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:14.599563 10054 raft_consensus.cc:2802] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.599673 10054 raft_consensus.cc:695] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 1 LEADER]: Becoming Leader. State: Replica: c0d8962738a24dff98cecf26cfc52d75, State: Running, Role: LEADER
I20250904 07:54:14.599886 10052 ts_tablet_manager.cc:1428] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250904 07:54:14.599886 10054 consensus_queue.cc:237] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } }
I20250904 07:54:14.601601 9897 catalog_manager.cc:5582] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0d8962738a24dff98cecf26cfc52d75 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "c0d8962738a24dff98cecf26cfc52d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d8962738a24dff98cecf26cfc52d75" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45233 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:14.607314 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:14.614363 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:14.614661 413 tablet_replica.cc:331] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75: stopping tablet replica
I20250904 07:54:14.614883 413 raft_consensus.cc:2241] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.615046 413 raft_consensus.cc:2270] T 3f3cf82ec55147f0bb55b7bf4688b021 P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.615875 413 tablet_replica.cc:331] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75: stopping tablet replica
I20250904 07:54:14.615978 413 raft_consensus.cc:2241] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.616124 413 raft_consensus.cc:2270] T 282d67aef15e428483cf403855c44e5c P c0d8962738a24dff98cecf26cfc52d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.620519 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:14.624415 413 master.cc:561] Master@127.0.103.126:37359 shutting down...
I20250904 07:54:14.632777 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:14.632988 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:14.633136 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 69c307c831f64e00b0debc8e2f1c9873: stopping tablet replica
I20250904 07:54:14.647110 413 master.cc:583] Master@127.0.103.126:37359 shutdown complete.
I20250904 07:54:14.653505 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35741
I20250904 07:54:14.654013 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:14.655984 10061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.656100 10062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.656574 10064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.656929 413 server_base.cc:1047] running on GCE node
I20250904 07:54:14.657145 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.657192 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.657232 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454657229 us; error 0 us; skew 500 ppm
I20250904 07:54:14.657385 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.658454 413 webserver.cc:480] Webserver started at http://127.0.103.126:36759/ using document root <none> and password file <none>
I20250904 07:54:14.658650 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.658711 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.658838 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.659433 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "d8c510e2673d48f9adacc6aae2700084"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.661968 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:54:14.663408 10069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.663725 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:14.663818 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d8c510e2673d48f9adacc6aae2700084"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.663921 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.685731 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.686193 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.697932 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35741
I20250904 07:54:14.697954 10130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35741 every 8 connection(s)
I20250904 07:54:14.699065 10131 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.703245 10131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084: Bootstrap starting.
I20250904 07:54:14.704341 10131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.705693 10131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084: No bootstrap required, opened a new log
I20250904 07:54:14.706264 10131 raft_consensus.cc:357] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER }
I20250904 07:54:14.706378 10131 raft_consensus.cc:383] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.706432 10131 raft_consensus.cc:738] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c510e2673d48f9adacc6aae2700084, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.706631 10131 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER }
I20250904 07:54:14.706737 10131 raft_consensus.cc:397] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:14.706794 10131 raft_consensus.cc:491] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:14.706869 10131 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:14.708686 10131 raft_consensus.cc:513] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER }
I20250904 07:54:14.708873 10131 leader_election.cc:304] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c510e2673d48f9adacc6aae2700084; no voters:
I20250904 07:54:14.709195 10131 leader_election.cc:290] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:14.709311 10134 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:14.709738 10134 raft_consensus.cc:695] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 1 LEADER]: Becoming Leader. State: Replica: d8c510e2673d48f9adacc6aae2700084, State: Running, Role: LEADER
I20250904 07:54:14.709954 10134 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER }
I20250904 07:54:14.710021 10131 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:14.710621 10136 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c510e2673d48f9adacc6aae2700084. Latest consensus state: current_term: 1 leader_uuid: "d8c510e2673d48f9adacc6aae2700084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER } }
I20250904 07:54:14.710772 10136 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:14.710574 10135 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c510e2673d48f9adacc6aae2700084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c510e2673d48f9adacc6aae2700084" member_type: VOTER } }
I20250904 07:54:14.711035 10135 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:14.711973 10139 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:14.713253 10139 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:14.714315 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:14.715865 10139 catalog_manager.cc:1349] Generated new cluster ID: 4bb3aeca405f4f44ad00f8b0b5147ac1
I20250904 07:54:14.715940 10139 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:14.724942 10139 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:14.725692 10139 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:14.741696 10139 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084: Generated new TSK 0
I20250904 07:54:14.741923 10139 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:14.746721 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:14.749228 10153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.749259 10152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.749749 10155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.750160 413 server_base.cc:1047] running on GCE node
I20250904 07:54:14.750361 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.750411 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.750445 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454750442 us; error 0 us; skew 500 ppm
I20250904 07:54:14.750550 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.751567 413 webserver.cc:480] Webserver started at http://127.0.103.65:46575/ using document root <none> and password file <none>
I20250904 07:54:14.751751 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.751809 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.751893 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.752444 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "349bfeade5d1480a9d7deede2b731a0d"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.754817 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:54:14.756191 10160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.756494 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:14.756587 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "349bfeade5d1480a9d7deede2b731a0d"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.756695 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.766691 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.767129 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.768589 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:14.768723 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.768883 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:14.768996 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.785035 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39771
I20250904 07:54:14.785060 10230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39771 every 8 connection(s)
I20250904 07:54:14.786370 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250904 07:54:14.786414 10231 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35741
I20250904 07:54:14.786535 10231 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:14.786825 10231 heartbeater.cc:507] Master 127.0.103.126:35741 requested a full tablet report, sending...
I20250904 07:54:14.787530 10086 ts_manager.cc:194] Registered new tserver with Master: 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65:39771)
I20250904 07:54:14.788627 10086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56870
W20250904 07:54:14.788650 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.789088 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.792917 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.793040 413 server_base.cc:1047] running on GCE node
I20250904 07:54:14.793326 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.793395 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.793432 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454793429 us; error 0 us; skew 500 ppm
I20250904 07:54:14.793570 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.794575 413 webserver.cc:480] Webserver started at http://127.0.103.66:36315/ using document root <none> and password file <none>
I20250904 07:54:14.794749 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.794806 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.794912 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.795446 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-1-root/instance:
uuid: "c4a5a199690e4c28958845ccf459eb0b"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.797590 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:54:14.798872 10241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.799177 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:14.799265 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-1-root
uuid: "c4a5a199690e4c28958845ccf459eb0b"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.799355 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.799834 10231 heartbeater.cc:499] Master 127.0.103.126:35741 was elected leader, sending a full tablet report...
I20250904 07:54:14.824069 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.824471 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.825923 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:14.826030 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.826120 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:14.826160 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.841354 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:45553
I20250904 07:54:14.841382 10310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:45553 every 8 connection(s)
I20250904 07:54:14.843014 10311 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35741
I20250904 07:54:14.843123 10311 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:14.843356 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250904 07:54:14.843389 10311 heartbeater.cc:507] Master 127.0.103.126:35741 requested a full tablet report, sending...
I20250904 07:54:14.844103 10085 ts_manager.cc:194] Registered new tserver with Master: c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553)
I20250904 07:54:14.845093 10085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56878
W20250904 07:54:14.845727 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:14.847150 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.849061 413 server_base.cc:1047] running on GCE node
W20250904 07:54:14.849834 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:14.850123 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:14.850178 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:14.850209 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972454850207 us; error 0 us; skew 500 ppm
I20250904 07:54:14.850294 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:14.851286 413 webserver.cc:480] Webserver started at http://127.0.103.67:37283/ using document root <none> and password file <none>
I20250904 07:54:14.851444 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:14.851495 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:14.851570 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:14.852063 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-2-root/instance:
uuid: "64d8c196d6994a65b6510bfa13f4eb68"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.854221 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:54:14.855598 10321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.855926 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:14.856019 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-2-root
uuid: "64d8c196d6994a65b6510bfa13f4eb68"
format_stamp: "Formatted at 2025-09-04 07:54:14 on dist-test-slave-ghlz"
I20250904 07:54:14.856098 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.KeepAlivePeriodically_KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers_1.1756972373064810-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:14.856251 10311 heartbeater.cc:499] Master 127.0.103.126:35741 was elected leader, sending a full tablet report...
I20250904 07:54:14.883193 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:14.883638 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:14.885147 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:14.885229 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.885380 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:14.885440 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:14.903273 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:39453
I20250904 07:54:14.903287 10390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:39453 every 8 connection(s)
I20250904 07:54:14.904724 10391 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35741
I20250904 07:54:14.904822 10391 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:14.905119 10391 heartbeater.cc:507] Master 127.0.103.126:35741 requested a full tablet report, sending...
I20250904 07:54:14.905777 10085 ts_manager.cc:194] Registered new tserver with Master: 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67:39453)
I20250904 07:54:14.906355 10085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56888
I20250904 07:54:14.906493 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.002287592s
I20250904 07:54:14.915555 10085 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56906:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250904 07:54:14.916221 10085 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table client-testtb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 07:54:14.917317 10391 heartbeater.cc:499] Master 127.0.103.126:35741 was elected leader, sending a full tablet report...
I20250904 07:54:14.946542 10195 tablet_service.cc:1468] Processing CreateTablet for tablet 20cfa6030d8c4d3084f97c69030cf208 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.947005 10195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20cfa6030d8c4d3084f97c69030cf208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.947295 10353 tablet_service.cc:1468] Processing CreateTablet for tablet 20cfa6030d8c4d3084f97c69030cf208 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.947660 10353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20cfa6030d8c4d3084f97c69030cf208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.950518 10270 tablet_service.cc:1468] Processing CreateTablet for tablet 31928204fb9644b29f47da47705e2c72 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.950565 10271 tablet_service.cc:1468] Processing CreateTablet for tablet 20cfa6030d8c4d3084f97c69030cf208 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.950878 10270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31928204fb9644b29f47da47705e2c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.951388 10269 tablet_service.cc:1468] Processing CreateTablet for tablet 7f92770ea13d4897bc66044b0a5ea903 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.951717 10269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f92770ea13d4897bc66044b0a5ea903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.946620 10194 tablet_service.cc:1468] Processing CreateTablet for tablet 31928204fb9644b29f47da47705e2c72 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.953272 10194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31928204fb9644b29f47da47705e2c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.954833 10192 tablet_service.cc:1468] Processing CreateTablet for tablet 7f92770ea13d4897bc66044b0a5ea903 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.955171 10192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f92770ea13d4897bc66044b0a5ea903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.957161 10271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20cfa6030d8c4d3084f97c69030cf208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.957237 10351 tablet_service.cc:1468] Processing CreateTablet for tablet 31928204fb9644b29f47da47705e2c72 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.957595 10351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31928204fb9644b29f47da47705e2c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.961189 10349 tablet_service.cc:1468] Processing CreateTablet for tablet 7f92770ea13d4897bc66044b0a5ea903 (DEFAULT_TABLE table=client-testtb [id=5bf0e2547a874b759e9da62592d4b1b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250904 07:54:14.961817 10349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f92770ea13d4897bc66044b0a5ea903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:14.971906 10407 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: Bootstrap starting.
I20250904 07:54:14.973217 10407 tablet_bootstrap.cc:654] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.974059 10405 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: Bootstrap starting.
I20250904 07:54:14.975304 10405 tablet_bootstrap.cc:654] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.976306 10406 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: Bootstrap starting.
I20250904 07:54:14.976792 10407 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: No bootstrap required, opened a new log
I20250904 07:54:14.977010 10407 ts_tablet_manager.cc:1397] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:54:14.978001 10407 raft_consensus.cc:357] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.978300 10407 raft_consensus.cc:383] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.978380 10407 raft_consensus.cc:738] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349bfeade5d1480a9d7deede2b731a0d, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.978224 10406 tablet_bootstrap.cc:654] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.978590 10407 consensus_queue.cc:260] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.978947 10405 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: No bootstrap required, opened a new log
I20250904 07:54:14.979125 10405 ts_tablet_manager.cc:1397] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250904 07:54:14.979331 10407 ts_tablet_manager.cc:1428] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:14.979705 10407 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: Bootstrap starting.
I20250904 07:54:14.979826 10405 raft_consensus.cc:357] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.979959 10405 raft_consensus.cc:383] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.980043 10405 raft_consensus.cc:738] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d8c196d6994a65b6510bfa13f4eb68, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.980252 10405 consensus_queue.cc:260] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.981724 10407 tablet_bootstrap.cc:654] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.983812 10405 ts_tablet_manager.cc:1428] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250904 07:54:14.983927 10407 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: No bootstrap required, opened a new log
I20250904 07:54:14.984091 10405 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: Bootstrap starting.
I20250904 07:54:14.984213 10407 ts_tablet_manager.cc:1397] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250904 07:54:14.985129 10407 raft_consensus.cc:357] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.985373 10407 raft_consensus.cc:383] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.985493 10407 raft_consensus.cc:738] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349bfeade5d1480a9d7deede2b731a0d, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.985764 10407 consensus_queue.cc:260] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.986241 10407 ts_tablet_manager.cc:1428] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:14.986529 10407 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: Bootstrap starting.
I20250904 07:54:14.986742 10406 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: No bootstrap required, opened a new log
I20250904 07:54:14.986855 10406 ts_tablet_manager.cc:1397] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250904 07:54:14.987634 10406 raft_consensus.cc:357] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.988077 10407 tablet_bootstrap.cc:654] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.988168 10406 raft_consensus.cc:383] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.988232 10406 raft_consensus.cc:738] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a5a199690e4c28958845ccf459eb0b, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.988077 10405 tablet_bootstrap.cc:654] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.989081 10406 consensus_queue.cc:260] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.989971 10407 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: No bootstrap required, opened a new log
I20250904 07:54:14.990126 10407 ts_tablet_manager.cc:1397] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250904 07:54:14.990181 10405 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: No bootstrap required, opened a new log
I20250904 07:54:14.990284 10405 ts_tablet_manager.cc:1397] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250904 07:54:14.990850 10405 raft_consensus.cc:357] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.990868 10407 raft_consensus.cc:357] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.991034 10405 raft_consensus.cc:383] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.991166 10407 raft_consensus.cc:383] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.991190 10405 raft_consensus.cc:738] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d8c196d6994a65b6510bfa13f4eb68, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.991246 10407 raft_consensus.cc:738] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349bfeade5d1480a9d7deede2b731a0d, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.991380 10405 consensus_queue.cc:260] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.991559 10407 consensus_queue.cc:260] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.991870 10405 ts_tablet_manager.cc:1428] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250904 07:54:14.992133 10405 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: Bootstrap starting.
I20250904 07:54:14.993345 10407 ts_tablet_manager.cc:1428] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:54:14.993558 10405 tablet_bootstrap.cc:654] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.994076 10406 ts_tablet_manager.cc:1428] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250904 07:54:14.994403 10406 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: Bootstrap starting.
I20250904 07:54:14.995934 10406 tablet_bootstrap.cc:654] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:14.997929 10406 tablet_bootstrap.cc:492] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: No bootstrap required, opened a new log
I20250904 07:54:14.997980 10405 tablet_bootstrap.cc:492] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: No bootstrap required, opened a new log
I20250904 07:54:14.998134 10406 ts_tablet_manager.cc:1397] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250904 07:54:14.998190 10405 ts_tablet_manager.cc:1397] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20250904 07:54:14.998708 10406 raft_consensus.cc:357] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.998921 10406 raft_consensus.cc:383] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.998884 10405 raft_consensus.cc:357] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.999042 10406 raft_consensus.cc:738] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a5a199690e4c28958845ccf459eb0b, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.999154 10405 raft_consensus.cc:383] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:14.999225 10405 raft_consensus.cc:738] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d8c196d6994a65b6510bfa13f4eb68, State: Initialized, Role: FOLLOWER
I20250904 07:54:14.999339 10406 consensus_queue.cc:260] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.999423 10405 consensus_queue.cc:260] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:14.999790 10406 ts_tablet_manager.cc:1428] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250904 07:54:15.000041 10405 ts_tablet_manager.cc:1428] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250904 07:54:15.000048 10406 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: Bootstrap starting.
I20250904 07:54:15.001428 10406 tablet_bootstrap.cc:654] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:15.002995 10406 tablet_bootstrap.cc:492] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: No bootstrap required, opened a new log
I20250904 07:54:15.003094 10406 ts_tablet_manager.cc:1397] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:15.003599 10406 raft_consensus.cc:357] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.003715 10406 raft_consensus.cc:383] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:15.003847 10406 raft_consensus.cc:738] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a5a199690e4c28958845ccf459eb0b, State: Initialized, Role: FOLLOWER
I20250904 07:54:15.004036 10406 consensus_queue.cc:260] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.004356 10406 ts_tablet_manager.cc:1428] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250904 07:54:15.041967 10410 raft_consensus.cc:491] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:15.042119 10410 raft_consensus.cc:513] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.042902 10410 leader_election.cc:290] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553), 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67:39453)
I20250904 07:54:15.048658 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31928204fb9644b29f47da47705e2c72" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b" is_pre_election: true
I20250904 07:54:15.048908 10283 raft_consensus.cc:2466] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 0.
I20250904 07:54:15.049290 10162 leader_election.cc:304] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, c4a5a199690e4c28958845ccf459eb0b; no voters:
I20250904 07:54:15.049532 10410 raft_consensus.cc:2802] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:54:15.049499 10366 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31928204fb9644b29f47da47705e2c72" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d8c196d6994a65b6510bfa13f4eb68" is_pre_election: true
I20250904 07:54:15.049611 10410 raft_consensus.cc:491] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:15.049672 10366 raft_consensus.cc:2466] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 0.
I20250904 07:54:15.049692 10410 raft_consensus.cc:3058] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.051863 10410 raft_consensus.cc:513] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.052414 10410 leader_election.cc:290] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 election: Requested vote from peers c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553), 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67:39453)
I20250904 07:54:15.052682 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31928204fb9644b29f47da47705e2c72" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b"
I20250904 07:54:15.052816 10283 raft_consensus.cc:3058] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.052805 10366 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31928204fb9644b29f47da47705e2c72" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d8c196d6994a65b6510bfa13f4eb68"
I20250904 07:54:15.053009 10366 raft_consensus.cc:3058] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.055068 10283 raft_consensus.cc:2466] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 1.
I20250904 07:54:15.055069 10366 raft_consensus.cc:2466] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 1.
I20250904 07:54:15.055444 10164 leader_election.cc:304] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, 64d8c196d6994a65b6510bfa13f4eb68, c4a5a199690e4c28958845ccf459eb0b; no voters:
I20250904 07:54:15.055754 10410 raft_consensus.cc:2802] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:15.056056 10410 raft_consensus.cc:695] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 1 LEADER]: Becoming Leader. State: Replica: 349bfeade5d1480a9d7deede2b731a0d, State: Running, Role: LEADER
I20250904 07:54:15.056258 10410 consensus_queue.cc:237] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.058300 10084 catalog_manager.cc:5582] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65). New cstate: current_term: 1 leader_uuid: "349bfeade5d1480a9d7deede2b731a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:15.062497 10412 raft_consensus.cc:491] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:15.062619 10412 raft_consensus.cc:513] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.063241 10412 leader_election.cc:290] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65:39771), c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553)
I20250904 07:54:15.068447 10206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20cfa6030d8c4d3084f97c69030cf208" candidate_uuid: "64d8c196d6994a65b6510bfa13f4eb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349bfeade5d1480a9d7deede2b731a0d" is_pre_election: true
I20250904 07:54:15.068514 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20cfa6030d8c4d3084f97c69030cf208" candidate_uuid: "64d8c196d6994a65b6510bfa13f4eb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b" is_pre_election: true
I20250904 07:54:15.068624 10206 raft_consensus.cc:2466] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d8c196d6994a65b6510bfa13f4eb68 in term 0.
I20250904 07:54:15.068706 10283 raft_consensus.cc:2466] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64d8c196d6994a65b6510bfa13f4eb68 in term 0.
I20250904 07:54:15.068979 10325 leader_election.cc:304] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, 64d8c196d6994a65b6510bfa13f4eb68; no voters:
I20250904 07:54:15.069236 10412 raft_consensus.cc:2802] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:54:15.069307 10412 raft_consensus.cc:491] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:15.069384 10412 raft_consensus.cc:3058] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.071380 10412 raft_consensus.cc:513] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.071930 10412 leader_election.cc:290] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [CANDIDATE]: Term 1 election: Requested vote from peers 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65:39771), c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553)
I20250904 07:54:15.072206 10206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20cfa6030d8c4d3084f97c69030cf208" candidate_uuid: "64d8c196d6994a65b6510bfa13f4eb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349bfeade5d1480a9d7deede2b731a0d"
I20250904 07:54:15.072331 10206 raft_consensus.cc:3058] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.073302 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20cfa6030d8c4d3084f97c69030cf208" candidate_uuid: "64d8c196d6994a65b6510bfa13f4eb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b"
I20250904 07:54:15.073427 10283 raft_consensus.cc:3058] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.074616 10206 raft_consensus.cc:2466] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d8c196d6994a65b6510bfa13f4eb68 in term 1.
I20250904 07:54:15.074944 10325 leader_election.cc:304] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, 64d8c196d6994a65b6510bfa13f4eb68; no voters:
I20250904 07:54:15.075183 10412 raft_consensus.cc:2802] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:15.075531 10412 raft_consensus.cc:695] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 LEADER]: Becoming Leader. State: Replica: 64d8c196d6994a65b6510bfa13f4eb68, State: Running, Role: LEADER
I20250904 07:54:15.075742 10283 raft_consensus.cc:2466] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64d8c196d6994a65b6510bfa13f4eb68 in term 1.
I20250904 07:54:15.075951 10412 consensus_queue.cc:237] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.078081 10084 catalog_manager.cc:5582] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "64d8c196d6994a65b6510bfa13f4eb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:15.179294 10410 raft_consensus.cc:491] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 07:54:15.179483 10410 raft_consensus.cc:513] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.180022 10410 leader_election.cc:290] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553), 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67:39453)
I20250904 07:54:15.180397 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f92770ea13d4897bc66044b0a5ea903" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b" is_pre_election: true
I20250904 07:54:15.180557 10283 raft_consensus.cc:2466] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 0.
I20250904 07:54:15.180907 10162 leader_election.cc:304] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, c4a5a199690e4c28958845ccf459eb0b; no voters:
I20250904 07:54:15.181175 10410 raft_consensus.cc:2802] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 07:54:15.181238 10410 raft_consensus.cc:491] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 07:54:15.181270 10366 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f92770ea13d4897bc66044b0a5ea903" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d8c196d6994a65b6510bfa13f4eb68" is_pre_election: true
I20250904 07:54:15.181283 10410 raft_consensus.cc:3058] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.181470 10366 raft_consensus.cc:2466] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 0.
I20250904 07:54:15.183498 10410 raft_consensus.cc:513] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.184064 10410 leader_election.cc:290] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 election: Requested vote from peers c4a5a199690e4c28958845ccf459eb0b (127.0.103.66:45553), 64d8c196d6994a65b6510bfa13f4eb68 (127.0.103.67:39453)
I20250904 07:54:15.184314 10283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f92770ea13d4897bc66044b0a5ea903" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4a5a199690e4c28958845ccf459eb0b"
I20250904 07:54:15.184454 10283 raft_consensus.cc:3058] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.184513 10366 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f92770ea13d4897bc66044b0a5ea903" candidate_uuid: "349bfeade5d1480a9d7deede2b731a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d8c196d6994a65b6510bfa13f4eb68"
I20250904 07:54:15.184634 10366 raft_consensus.cc:3058] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:15.187240 10366 raft_consensus.cc:2466] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 1.
I20250904 07:54:15.187281 10283 raft_consensus.cc:2466] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349bfeade5d1480a9d7deede2b731a0d in term 1.
I20250904 07:54:15.187644 10164 leader_election.cc:304] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349bfeade5d1480a9d7deede2b731a0d, 64d8c196d6994a65b6510bfa13f4eb68; no voters:
I20250904 07:54:15.187886 10410 raft_consensus.cc:2802] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:15.188007 10410 raft_consensus.cc:695] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 1 LEADER]: Becoming Leader. State: Replica: 349bfeade5d1480a9d7deede2b731a0d, State: Running, Role: LEADER
I20250904 07:54:15.188231 10410 consensus_queue.cc:237] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } }
I20250904 07:54:15.190208 10084 catalog_manager.cc:5582] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65). New cstate: current_term: 1 leader_uuid: "349bfeade5d1480a9d7deede2b731a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 } health_report { overall_health: UNKNOWN } } }
I20250904 07:54:15.407357 10366 raft_consensus.cc:1273] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Refusing update from remote peer 349bfeade5d1480a9d7deede2b731a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.407511 10283 raft_consensus.cc:1273] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Refusing update from remote peer 349bfeade5d1480a9d7deede2b731a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.408001 10285 raft_consensus.cc:1273] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Refusing update from remote peer 64d8c196d6994a65b6510bfa13f4eb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.408141 10206 raft_consensus.cc:1273] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Refusing update from remote peer 64d8c196d6994a65b6510bfa13f4eb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.408382 10410 consensus_queue.cc:1035] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:15.408628 10421 consensus_queue.cc:1035] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:15.408651 10424 consensus_queue.cc:1035] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:15.409067 10412 consensus_queue.cc:1035] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "349bfeade5d1480a9d7deede2b731a0d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:15.498186 10366 raft_consensus.cc:1273] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Refusing update from remote peer 349bfeade5d1480a9d7deede2b731a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.498801 10421 consensus_queue.cc:1035] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d8c196d6994a65b6510bfa13f4eb68" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:15.500463 10283 raft_consensus.cc:1273] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Refusing update from remote peer 349bfeade5d1480a9d7deede2b731a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 07:54:15.501230 10421 consensus_queue.cc:1035] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5a199690e4c28958845ccf459eb0b" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 07:54:40.658226 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20250904 07:54:40.671540 10325 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250904 07:54:40.677651 10325 consensus_peers.cc:489] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 -> Peer 349bfeade5d1480a9d7deede2b731a0d (127.0.103.65:39771): Couldn't send request to peer 349bfeade5d1480a9d7deede2b731a0d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250904 07:54:40.700846 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:40.701133 413 tablet_replica.cc:331] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d: stopping tablet replica
I20250904 07:54:40.701339 413 raft_consensus.cc:2241] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.701517 413 raft_consensus.cc:2270] T 20cfa6030d8c4d3084f97c69030cf208 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.702100 413 tablet_replica.cc:331] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d: stopping tablet replica
I20250904 07:54:40.702204 413 raft_consensus.cc:2241] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:40.702425 413 raft_consensus.cc:2270] T 7f92770ea13d4897bc66044b0a5ea903 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.702874 413 tablet_replica.cc:331] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d: stopping tablet replica
I20250904 07:54:40.702965 413 raft_consensus.cc:2241] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:40.703153 413 raft_consensus.cc:2270] T 31928204fb9644b29f47da47705e2c72 P 349bfeade5d1480a9d7deede2b731a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.716898 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:40.721066 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250904 07:54:40.729858 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:40.730062 413 tablet_replica.cc:331] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b: stopping tablet replica
I20250904 07:54:40.730232 413 raft_consensus.cc:2241] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.730438 413 raft_consensus.cc:2270] T 7f92770ea13d4897bc66044b0a5ea903 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.730995 413 tablet_replica.cc:331] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b: stopping tablet replica
I20250904 07:54:40.731125 413 raft_consensus.cc:2241] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.731274 413 raft_consensus.cc:2270] T 31928204fb9644b29f47da47705e2c72 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.731773 413 tablet_replica.cc:331] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b: stopping tablet replica
I20250904 07:54:40.731899 413 raft_consensus.cc:2241] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.732065 413 raft_consensus.cc:2270] T 20cfa6030d8c4d3084f97c69030cf208 P c4a5a199690e4c28958845ccf459eb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.746840 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250904 07:54:40.750578 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250904 07:54:40.759471 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:40.759657 413 tablet_replica.cc:331] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68: stopping tablet replica
I20250904 07:54:40.759819 413 raft_consensus.cc:2241] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.759995 413 raft_consensus.cc:2270] T 31928204fb9644b29f47da47705e2c72 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.760536 413 tablet_replica.cc:331] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68: stopping tablet replica
I20250904 07:54:40.760665 413 raft_consensus.cc:2241] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:40.760932 413 raft_consensus.cc:2270] T 20cfa6030d8c4d3084f97c69030cf208 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.761423 413 tablet_replica.cc:331] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68: stopping tablet replica
I20250904 07:54:40.761545 413 raft_consensus.cc:2241] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 07:54:40.761689 413 raft_consensus.cc:2270] T 7f92770ea13d4897bc66044b0a5ea903 P 64d8c196d6994a65b6510bfa13f4eb68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.775841 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250904 07:54:40.779616 413 master.cc:561] Master@127.0.103.126:35741 shutting down...
I20250904 07:54:40.788883 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:40.789067 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:40.789149 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d8c510e2673d48f9adacc6aae2700084: stopping tablet replica
I20250904 07:54:40.810060 413 master.cc:583] Master@127.0.103.126:35741 shutdown complete.
[ OK ] KeepAlivePeriodically/KeepAlivePeriodicallyTest.TestScannerKeepAlivePeriodicallyCrossServers/1 (26471 ms)
[----------] 1 test from KeepAlivePeriodically/KeepAlivePeriodicallyTest (26471 ms total)
[----------] 1 test from Params/LatestObservedTimestampParamTest
[ RUN ] Params/LatestObservedTimestampParamTest.Test/1
I20250904 07:54:40.822003 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41401
I20250904 07:54:40.822464 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:40.824430 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:40.824440 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:40.824940 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:40.825313 413 server_base.cc:1047] running on GCE node
I20250904 07:54:40.825506 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:40.825551 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:40.825591 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972480825587 us; error 0 us; skew 500 ppm
I20250904 07:54:40.825733 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:40.826766 413 webserver.cc:480] Webserver started at http://127.0.103.126:34675/ using document root <none> and password file <none>
I20250904 07:54:40.826969 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:40.827035 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:40.827152 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:40.827688 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "608901e2472f46688541b1f88fe11286"
format_stamp: "Formatted at 2025-09-04 07:54:40 on dist-test-slave-ghlz"
I20250904 07:54:40.829948 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:54:40.831276 10466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:40.831557 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:40.831653 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "608901e2472f46688541b1f88fe11286"
format_stamp: "Formatted at 2025-09-04 07:54:40 on dist-test-slave-ghlz"
I20250904 07:54:40.831749 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:40.859901 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:40.860317 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:40.871399 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41401
I20250904 07:54:40.871417 10527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41401 every 8 connection(s)
I20250904 07:54:40.872526 10528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:40.876487 10528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286: Bootstrap starting.
I20250904 07:54:40.877568 10528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:40.878787 10528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286: No bootstrap required, opened a new log
I20250904 07:54:40.879272 10528 raft_consensus.cc:357] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER }
I20250904 07:54:40.879375 10528 raft_consensus.cc:383] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:40.879410 10528 raft_consensus.cc:738] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608901e2472f46688541b1f88fe11286, State: Initialized, Role: FOLLOWER
I20250904 07:54:40.879567 10528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER }
I20250904 07:54:40.879678 10528 raft_consensus.cc:397] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:40.879711 10528 raft_consensus.cc:491] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:40.879760 10528 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:40.881466 10528 raft_consensus.cc:513] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER }
I20250904 07:54:40.881605 10528 leader_election.cc:304] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608901e2472f46688541b1f88fe11286; no voters:
I20250904 07:54:40.881903 10528 leader_election.cc:290] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:40.882017 10531 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:40.882436 10531 raft_consensus.cc:695] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 1 LEADER]: Becoming Leader. State: Replica: 608901e2472f46688541b1f88fe11286, State: Running, Role: LEADER
I20250904 07:54:40.882591 10528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:40.882620 10531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER }
I20250904 07:54:40.883240 10533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608901e2472f46688541b1f88fe11286. Latest consensus state: current_term: 1 leader_uuid: "608901e2472f46688541b1f88fe11286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER } }
I20250904 07:54:40.883375 10533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:40.883404 10532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608901e2472f46688541b1f88fe11286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608901e2472f46688541b1f88fe11286" member_type: VOTER } }
I20250904 07:54:40.883630 10532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:40.884102 10536 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:40.885499 10536 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:40.886745 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:40.887812 10536 catalog_manager.cc:1349] Generated new cluster ID: 26d1859561404cebb64c153714cd1cf0
I20250904 07:54:40.887876 10536 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:40.902724 10536 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:40.903378 10536 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:40.913522 10536 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286: Generated new TSK 0
I20250904 07:54:40.913681 10536 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:40.918989 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:40.921253 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:40.921254 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:40.921779 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:40.921924 413 server_base.cc:1047] running on GCE node
I20250904 07:54:40.922176 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:40.922246 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:40.922291 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972480922286 us; error 0 us; skew 500 ppm
I20250904 07:54:40.922428 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:40.923439 413 webserver.cc:480] Webserver started at http://127.0.103.65:32887/ using document root <none> and password file <none>
I20250904 07:54:40.923614 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:40.923671 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:40.923784 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:40.924268 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "4101f06df6254cc28d636e1c49d779f4"
format_stamp: "Formatted at 2025-09-04 07:54:40 on dist-test-slave-ghlz"
I20250904 07:54:40.926369 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250904 07:54:40.927660 10557 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:40.927999 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:40.928092 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "4101f06df6254cc28d636e1c49d779f4"
format_stamp: "Formatted at 2025-09-04 07:54:40 on dist-test-slave-ghlz"
I20250904 07:54:40.928189 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_LatestObservedTimestampParamTest.Test_1.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:40.937923 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:40.938294 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:40.939625 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:40.939745 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:40.939855 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:40.939944 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:40.951716 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:42845
I20250904 07:54:40.951753 10625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:42845 every 8 connection(s)
I20250904 07:54:40.953323 10627 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41401
I20250904 07:54:40.953431 10627 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:40.953739 10627 heartbeater.cc:507] Master 127.0.103.126:41401 requested a full tablet report, sending...
I20250904 07:54:40.954478 10483 ts_manager.cc:194] Registered new tserver with Master: 4101f06df6254cc28d636e1c49d779f4 (127.0.103.65:42845)
I20250904 07:54:40.955011 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00227902s
I20250904 07:54:40.955735 10483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32772
I20250904 07:54:40.966650 10483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32796:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:54:40.967195 10627 heartbeater.cc:499] Master 127.0.103.126:41401 was elected leader, sending a full tablet report...
I20250904 07:54:40.978183 10588 tablet_service.cc:1468] Processing CreateTablet for tablet a0334c9461f7479aa225881eef5f9619 (DEFAULT_TABLE table=client-testtb [id=e9fd345598a24ddd8a69de41c4764d9d]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:54:40.978243 10587 tablet_service.cc:1468] Processing CreateTablet for tablet 8ea224081db344caaca666c1b49f4e52 (DEFAULT_TABLE table=client-testtb [id=e9fd345598a24ddd8a69de41c4764d9d]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:54:40.978615 10588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0334c9461f7479aa225881eef5f9619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:40.978849 10587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea224081db344caaca666c1b49f4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:40.983729 10638 tablet_bootstrap.cc:492] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: Bootstrap starting.
I20250904 07:54:40.984731 10638 tablet_bootstrap.cc:654] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:40.986140 10638 tablet_bootstrap.cc:492] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: No bootstrap required, opened a new log
I20250904 07:54:40.986254 10638 ts_tablet_manager.cc:1397] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:40.986742 10638 raft_consensus.cc:357] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.986864 10638 raft_consensus.cc:383] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:40.986917 10638 raft_consensus.cc:738] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4101f06df6254cc28d636e1c49d779f4, State: Initialized, Role: FOLLOWER
I20250904 07:54:40.987231 10638 consensus_queue.cc:260] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.987357 10638 raft_consensus.cc:397] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:40.987422 10638 raft_consensus.cc:491] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:40.987495 10638 raft_consensus.cc:3058] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:40.989293 10638 raft_consensus.cc:513] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.989457 10638 leader_election.cc:304] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4101f06df6254cc28d636e1c49d779f4; no voters:
I20250904 07:54:40.989823 10638 leader_election.cc:290] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:40.989956 10640 raft_consensus.cc:2802] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:40.990521 10638 ts_tablet_manager.cc:1428] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250904 07:54:40.990795 10638 tablet_bootstrap.cc:492] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: Bootstrap starting.
I20250904 07:54:40.991986 10638 tablet_bootstrap.cc:654] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:40.993031 10640 raft_consensus.cc:695] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4101f06df6254cc28d636e1c49d779f4, State: Running, Role: LEADER
I20250904 07:54:40.993240 10640 consensus_queue.cc:237] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.993304 10638 tablet_bootstrap.cc:492] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: No bootstrap required, opened a new log
I20250904 07:54:40.993391 10638 ts_tablet_manager.cc:1397] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:40.994040 10638 raft_consensus.cc:357] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.994177 10638 raft_consensus.cc:383] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:40.994243 10638 raft_consensus.cc:738] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4101f06df6254cc28d636e1c49d779f4, State: Initialized, Role: FOLLOWER
I20250904 07:54:40.994415 10638 consensus_queue.cc:260] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.994549 10638 raft_consensus.cc:397] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:40.994609 10638 raft_consensus.cc:491] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:40.994676 10638 raft_consensus.cc:3058] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:40.995087 10483 catalog_manager.cc:5582] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4101f06df6254cc28d636e1c49d779f4 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4101f06df6254cc28d636e1c49d779f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:40.996610 10638 raft_consensus.cc:513] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.996800 10638 leader_election.cc:304] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4101f06df6254cc28d636e1c49d779f4; no voters:
I20250904 07:54:40.996976 10638 leader_election.cc:290] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:40.997051 10640 raft_consensus.cc:2802] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:40.997205 10640 raft_consensus.cc:695] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4101f06df6254cc28d636e1c49d779f4, State: Running, Role: LEADER
I20250904 07:54:40.997424 10638 ts_tablet_manager.cc:1428] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250904 07:54:40.997450 10640 consensus_queue.cc:237] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } }
I20250904 07:54:40.999276 10483 catalog_manager.cc:5582] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4101f06df6254cc28d636e1c49d779f4 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "4101f06df6254cc28d636e1c49d779f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4101f06df6254cc28d636e1c49d779f4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 42845 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:41.037052 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:41.044929 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:41.045207 413 tablet_replica.cc:331] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4: stopping tablet replica
I20250904 07:54:41.045378 413 raft_consensus.cc:2241] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:41.045529 413 raft_consensus.cc:2270] T 8ea224081db344caaca666c1b49f4e52 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:41.046254 413 tablet_replica.cc:331] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4: stopping tablet replica
I20250904 07:54:41.046378 413 raft_consensus.cc:2241] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:41.046523 413 raft_consensus.cc:2270] T a0334c9461f7479aa225881eef5f9619 P 4101f06df6254cc28d636e1c49d779f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:41.060520 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:41.064534 413 master.cc:561] Master@127.0.103.126:41401 shutting down...
I20250904 07:54:41.071820 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:41.072018 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:41.072145 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 608901e2472f46688541b1f88fe11286: stopping tablet replica
I20250904 07:54:41.075901 413 master.cc:583] Master@127.0.103.126:41401 shutdown complete.
[ OK ] Params/LatestObservedTimestampParamTest.Test/1 (264 ms)
[----------] 1 test from Params/LatestObservedTimestampParamTest (264 ms total)
[----------] 1 test from ServiceUnavailableRetryClientTest
[ RUN ] ServiceUnavailableRetryClientTest.CreateTable/0
I20250904 07:54:41.086500 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35083
I20250904 07:54:41.086891 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:41.088748 10655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:41.088819 10656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:41.089265 10658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:41.090191 413 server_base.cc:1047] running on GCE node
I20250904 07:54:41.090394 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:41.090445 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:41.090471 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972481090465 us; error 0 us; skew 500 ppm
I20250904 07:54:41.090602 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:41.091703 413 webserver.cc:480] Webserver started at http://127.0.103.126:39355/ using document root <none> and password file <none>
I20250904 07:54:41.091886 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:41.091939 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:41.092041 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:41.092592 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "d6265610140644088276183d92f6b718"
format_stamp: "Formatted at 2025-09-04 07:54:41 on dist-test-slave-ghlz"
I20250904 07:54:41.094842 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250904 07:54:41.096102 10663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:41.096392 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:41.096474 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "d6265610140644088276183d92f6b718"
format_stamp: "Formatted at 2025-09-04 07:54:41 on dist-test-slave-ghlz"
I20250904 07:54:41.096571 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:41.110926 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:41.111428 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:41.123006 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35083
I20250904 07:54:41.123034 10724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35083 every 8 connection(s)
I20250904 07:54:41.124054 10725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:41.128182 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718: Bootstrap starting.
I20250904 07:54:41.129266 10725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:41.130574 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718: No bootstrap required, opened a new log
I20250904 07:54:41.131104 10725 raft_consensus.cc:357] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER }
I20250904 07:54:41.131212 10725 raft_consensus.cc:383] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:41.131271 10725 raft_consensus.cc:738] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6265610140644088276183d92f6b718, State: Initialized, Role: FOLLOWER
I20250904 07:54:41.131453 10725 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER }
I20250904 07:54:41.131546 10725 raft_consensus.cc:397] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:41.131596 10725 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:41.131667 10725 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:41.133499 10725 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER }
I20250904 07:54:41.133647 10725 leader_election.cc:304] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6265610140644088276183d92f6b718; no voters:
I20250904 07:54:41.133993 10725 leader_election.cc:290] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:41.134104 10728 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:41.134485 10728 raft_consensus.cc:695] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 1 LEADER]: Becoming Leader. State: Replica: d6265610140644088276183d92f6b718, State: Running, Role: LEADER
I20250904 07:54:41.134682 10728 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER }
I20250904 07:54:41.134756 10725 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:41.135452 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6265610140644088276183d92f6b718. Latest consensus state: current_term: 1 leader_uuid: "d6265610140644088276183d92f6b718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER } }
I20250904 07:54:41.135654 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:41.135517 10729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6265610140644088276183d92f6b718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6265610140644088276183d92f6b718" member_type: VOTER } }
I20250904 07:54:41.135916 10729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:41.136616 10733 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:41.137688 10733 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:41.139431 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:41.140681 10733 catalog_manager.cc:1349] Generated new cluster ID: f538376b046e413ea1b202592b982278
I20250904 07:54:41.140764 10733 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:41.149492 10733 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:41.150172 10733 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:41.166911 10733 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718: Generated new TSK 0
I20250904 07:54:41.167153 10733 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:41.171825 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:41.174006 10746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:41.174067 10747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:41.174526 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:41.174687 413 server_base.cc:1047] running on GCE node
I20250904 07:54:41.174885 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:41.174932 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:41.174957 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972481174954 us; error 0 us; skew 500 ppm
I20250904 07:54:41.175088 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:41.176117 413 webserver.cc:480] Webserver started at http://127.0.103.65:42099/ using document root <none> and password file <none>
I20250904 07:54:41.176306 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:41.176363 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:41.176476 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:41.177017 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "5caed26b3f6a412f88cebc62edfd0a71"
format_stamp: "Formatted at 2025-09-04 07:54:41 on dist-test-slave-ghlz"
I20250904 07:54:41.179072 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:41.180315 10754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:41.180585 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:41.180675 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "5caed26b3f6a412f88cebc62edfd0a71"
format_stamp: "Formatted at 2025-09-04 07:54:41 on dist-test-slave-ghlz"
I20250904 07:54:41.180760 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.ServiceUnavailableRetryClientTest.CreateTable_0.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:41.186329 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:41.186687 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:41.188489 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:41.188547 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:41.188652 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:41.188691 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:41.206270 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45635
I20250904 07:54:41.206301 10824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45635 every 8 connection(s)
I20250904 07:54:41.207618 10825 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35083
I20250904 07:54:41.207788 10825 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:41.208088 10825 heartbeater.cc:507] Master 127.0.103.126:35083 requested a full tablet report, sending...
I20250904 07:54:41.208802 10680 ts_manager.cc:194] Registered new tserver with Master: 5caed26b3f6a412f88cebc62edfd0a71 (127.0.103.65:45635)
I20250904 07:54:41.209396 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002266268s
I20250904 07:54:41.209802 10680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46134
I20250904 07:54:41.218151 10680 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46152:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:54:41.220927 10825 heartbeater.cc:499] Master 127.0.103.126:35083 was elected leader, sending a full tablet report...
I20250904 07:54:41.229372 10782 tablet_service.cc:1468] Processing CreateTablet for tablet 267165d590a64bfd80a71a676da8466e (DEFAULT_TABLE table=client-testtb [id=ac6a5303fc834808a8daa575f28a71cf]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:54:41.229429 10785 tablet_service.cc:1468] Processing CreateTablet for tablet 88c20a35be804a34996724f401da0d8d (DEFAULT_TABLE table=client-testtb [id=ac6a5303fc834808a8daa575f28a71cf]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:54:41.229799 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267165d590a64bfd80a71a676da8466e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:41.230005 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88c20a35be804a34996724f401da0d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:41.235059 10835 tablet_bootstrap.cc:492] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: Bootstrap starting.
I20250904 07:54:41.236439 10835 tablet_bootstrap.cc:654] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:41.238195 10835 tablet_bootstrap.cc:492] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: No bootstrap required, opened a new log
I20250904 07:54:41.238332 10835 ts_tablet_manager.cc:1397] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:54:41.239079 10835 raft_consensus.cc:357] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.239255 10835 raft_consensus.cc:383] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:41.239317 10835 raft_consensus.cc:738] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caed26b3f6a412f88cebc62edfd0a71, State: Initialized, Role: FOLLOWER
I20250904 07:54:41.239534 10835 consensus_queue.cc:260] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.239698 10835 raft_consensus.cc:397] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:41.239768 10835 raft_consensus.cc:491] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:41.239850 10835 raft_consensus.cc:3058] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:41.242244 10835 raft_consensus.cc:513] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.242399 10835 leader_election.cc:304] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5caed26b3f6a412f88cebc62edfd0a71; no voters:
I20250904 07:54:41.242728 10835 leader_election.cc:290] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:41.242836 10837 raft_consensus.cc:2802] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:41.243268 10837 raft_consensus.cc:695] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 LEADER]: Becoming Leader. State: Replica: 5caed26b3f6a412f88cebc62edfd0a71, State: Running, Role: LEADER
I20250904 07:54:41.243386 10835 ts_tablet_manager.cc:1428] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20250904 07:54:41.243490 10837 consensus_queue.cc:237] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.243606 10835 tablet_bootstrap.cc:492] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: Bootstrap starting.
I20250904 07:54:41.244930 10835 tablet_bootstrap.cc:654] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:41.245226 10679 catalog_manager.cc:5582] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5caed26b3f6a412f88cebc62edfd0a71 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5caed26b3f6a412f88cebc62edfd0a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:41.247375 10835 tablet_bootstrap.cc:492] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: No bootstrap required, opened a new log
I20250904 07:54:41.247488 10835 ts_tablet_manager.cc:1397] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250904 07:54:41.248083 10835 raft_consensus.cc:357] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.248217 10835 raft_consensus.cc:383] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:41.248286 10835 raft_consensus.cc:738] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caed26b3f6a412f88cebc62edfd0a71, State: Initialized, Role: FOLLOWER
I20250904 07:54:41.248454 10835 consensus_queue.cc:260] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.248594 10835 raft_consensus.cc:397] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:41.248651 10835 raft_consensus.cc:491] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:41.248713 10835 raft_consensus.cc:3058] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:41.250497 10835 raft_consensus.cc:513] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.250653 10835 leader_election.cc:304] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5caed26b3f6a412f88cebc62edfd0a71; no voters:
I20250904 07:54:41.250797 10835 leader_election.cc:290] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:41.250852 10837 raft_consensus.cc:2802] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:41.250954 10837 raft_consensus.cc:695] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 LEADER]: Becoming Leader. State: Replica: 5caed26b3f6a412f88cebc62edfd0a71, State: Running, Role: LEADER
I20250904 07:54:41.251144 10835 ts_tablet_manager.cc:1428] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 07:54:41.251132 10837 consensus_queue.cc:237] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } }
I20250904 07:54:41.252676 10679 catalog_manager.cc:5582] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5caed26b3f6a412f88cebc62edfd0a71 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5caed26b3f6a412f88cebc62edfd0a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5caed26b3f6a412f88cebc62edfd0a71" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45635 } health_report { overall_health: HEALTHY } } }
W20250904 07:54:41.261734 10829 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.103.126:35083)
I20250904 07:54:43.261634 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:43.270028 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:43.270257 413 tablet_replica.cc:331] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71: stopping tablet replica
I20250904 07:54:43.270445 413 raft_consensus.cc:2241] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.270596 413 raft_consensus.cc:2270] T 267165d590a64bfd80a71a676da8466e P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.271122 413 tablet_replica.cc:331] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71: stopping tablet replica
I20250904 07:54:43.271225 413 raft_consensus.cc:2241] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.271332 413 raft_consensus.cc:2270] T 88c20a35be804a34996724f401da0d8d P 5caed26b3f6a412f88cebc62edfd0a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.275441 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:43.279264 413 master.cc:561] Master@127.0.103.126:35083 shutting down...
I20250904 07:54:43.286590 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.286760 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.286840 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d6265610140644088276183d92f6b718: stopping tablet replica
I20250904 07:54:43.299499 413 master.cc:583] Master@127.0.103.126:35083 shutdown complete.
[ OK ] ServiceUnavailableRetryClientTest.CreateTable/0 (2228 ms)
[----------] 1 test from ServiceUnavailableRetryClientTest (2228 ms total)
[----------] 2 tests from Params/ClientTestImmutableColumn
[ RUN ] Params/ClientTestImmutableColumn.TestUpdateIgnore/2
I20250904 07:54:43.315425 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37081
I20250904 07:54:43.315838 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:43.317883 10846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.317936 10845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.318394 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:43.318562 413 server_base.cc:1047] running on GCE node
I20250904 07:54:43.318825 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:43.318897 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:43.318941 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972483318936 us; error 0 us; skew 500 ppm
I20250904 07:54:43.319084 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:43.320164 413 webserver.cc:480] Webserver started at http://127.0.103.126:42929/ using document root <none> and password file <none>
I20250904 07:54:43.320353 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:43.320412 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:43.320525 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:43.321091 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "1a93f862923443648a47d479f02438b3"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.323216 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:43.324463 10853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.324749 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250904 07:54:43.324842 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "1a93f862923443648a47d479f02438b3"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.324991 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:43.334450 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:43.334810 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:43.346371 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37081
I20250904 07:54:43.346395 10914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:37081 every 8 connection(s)
I20250904 07:54:43.348294 10915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.352259 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3: Bootstrap starting.
I20250904 07:54:43.353343 10915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.354590 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3: No bootstrap required, opened a new log
I20250904 07:54:43.355126 10915 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER }
I20250904 07:54:43.355243 10915 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.355283 10915 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a93f862923443648a47d479f02438b3, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.355415 10915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER }
I20250904 07:54:43.355499 10915 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.355530 10915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.355584 10915 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.357326 10915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER }
I20250904 07:54:43.357479 10915 leader_election.cc:304] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a93f862923443648a47d479f02438b3; no voters:
I20250904 07:54:43.357774 10915 leader_election.cc:290] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.357872 10918 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.358336 10918 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a93f862923443648a47d479f02438b3, State: Running, Role: LEADER
I20250904 07:54:43.358549 10918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER }
I20250904 07:54:43.358642 10915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:43.359211 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a93f862923443648a47d479f02438b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER } }
I20250904 07:54:43.359266 10920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a93f862923443648a47d479f02438b3. Latest consensus state: current_term: 1 leader_uuid: "1a93f862923443648a47d479f02438b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93f862923443648a47d479f02438b3" member_type: VOTER } }
I20250904 07:54:43.359440 10920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:43.359474 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:43.360986 10923 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:43.362573 10923 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:43.363380 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:43.365460 10923 catalog_manager.cc:1349] Generated new cluster ID: b0f6275f82c943c9a8f8f0c2f52b0730
I20250904 07:54:43.365521 10923 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:43.376021 10923 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:43.376705 10923 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:43.383489 10923 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3: Generated new TSK 0
I20250904 07:54:43.383682 10923 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:43.396080 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:43.398367 10936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.398572 10937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.398836 10939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:43.399058 413 server_base.cc:1047] running on GCE node
I20250904 07:54:43.399294 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:43.399350 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:43.399394 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972483399390 us; error 0 us; skew 500 ppm
I20250904 07:54:43.399530 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:43.400544 413 webserver.cc:480] Webserver started at http://127.0.103.65:33161/ using document root <none> and password file <none>
I20250904 07:54:43.400730 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:43.400792 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:43.400928 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:43.401466 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "9ce532035e8e4611a15e5e39098b9006"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.403661 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250904 07:54:43.405190 10944 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.405472 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250904 07:54:43.405565 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "9ce532035e8e4611a15e5e39098b9006"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.405651 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpdateIgnore_2.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:43.413924 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:43.414312 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:43.415673 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:43.415751 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.415830 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:43.415910 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.430850 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40613
I20250904 07:54:43.430871 11014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40613 every 8 connection(s)
I20250904 07:54:43.432365 11015 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37081
I20250904 07:54:43.432483 11015 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:43.432745 11015 heartbeater.cc:507] Master 127.0.103.126:37081 requested a full tablet report, sending...
I20250904 07:54:43.433483 10870 ts_manager.cc:194] Registered new tserver with Master: 9ce532035e8e4611a15e5e39098b9006 (127.0.103.65:40613)
I20250904 07:54:43.434180 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002244214s
I20250904 07:54:43.434393 10870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44668
I20250904 07:54:43.443197 10870 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44688:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "imm_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: true
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:54:43.445453 11015 heartbeater.cc:499] Master 127.0.103.126:37081 was elected leader, sending a full tablet report...
I20250904 07:54:43.454470 10975 tablet_service.cc:1468] Processing CreateTablet for tablet 1c725f572ed24ed18c65c57326a4cb3c (DEFAULT_TABLE table=client-testtb [id=0d3c4ddfffb140bfa62daec67582ee27]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:54:43.454512 10974 tablet_service.cc:1468] Processing CreateTablet for tablet 193a9d1de28a4dbd95a2e235a04ac533 (DEFAULT_TABLE table=client-testtb [id=0d3c4ddfffb140bfa62daec67582ee27]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:54:43.454856 10975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c725f572ed24ed18c65c57326a4cb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.455072 10974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193a9d1de28a4dbd95a2e235a04ac533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.462121 11025 tablet_bootstrap.cc:492] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: Bootstrap starting.
I20250904 07:54:43.463266 11025 tablet_bootstrap.cc:654] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.464457 11025 tablet_bootstrap.cc:492] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: No bootstrap required, opened a new log
I20250904 07:54:43.464573 11025 ts_tablet_manager.cc:1397] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:43.465116 11025 raft_consensus.cc:357] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.465227 11025 raft_consensus.cc:383] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.465281 11025 raft_consensus.cc:738] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce532035e8e4611a15e5e39098b9006, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.465451 11025 consensus_queue.cc:260] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.465570 11025 raft_consensus.cc:397] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.465631 11025 raft_consensus.cc:491] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.465708 11025 raft_consensus.cc:3058] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.467860 11025 raft_consensus.cc:513] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.468056 11025 leader_election.cc:304] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ce532035e8e4611a15e5e39098b9006; no voters:
I20250904 07:54:43.468436 11025 leader_election.cc:290] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.468545 11027 raft_consensus.cc:2802] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.469146 11025 ts_tablet_manager.cc:1428] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250904 07:54:43.469177 11027 raft_consensus.cc:695] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 1 LEADER]: Becoming Leader. State: Replica: 9ce532035e8e4611a15e5e39098b9006, State: Running, Role: LEADER
I20250904 07:54:43.469381 11025 tablet_bootstrap.cc:492] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: Bootstrap starting.
I20250904 07:54:43.469404 11027 consensus_queue.cc:237] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.470423 11025 tablet_bootstrap.cc:654] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.471215 10870 catalog_manager.cc:5582] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ce532035e8e4611a15e5e39098b9006 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9ce532035e8e4611a15e5e39098b9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:43.471809 11025 tablet_bootstrap.cc:492] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: No bootstrap required, opened a new log
I20250904 07:54:43.471918 11025 ts_tablet_manager.cc:1397] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250904 07:54:43.472465 11025 raft_consensus.cc:357] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.472577 11025 raft_consensus.cc:383] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.472632 11025 raft_consensus.cc:738] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce532035e8e4611a15e5e39098b9006, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.472790 11025 consensus_queue.cc:260] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.472975 11025 raft_consensus.cc:397] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.473039 11025 raft_consensus.cc:491] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.473105 11025 raft_consensus.cc:3058] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.474839 11025 raft_consensus.cc:513] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.474983 11025 leader_election.cc:304] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ce532035e8e4611a15e5e39098b9006; no voters:
I20250904 07:54:43.475090 11025 leader_election.cc:290] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.475183 11027 raft_consensus.cc:2802] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.475314 11027 raft_consensus.cc:695] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 1 LEADER]: Becoming Leader. State: Replica: 9ce532035e8e4611a15e5e39098b9006, State: Running, Role: LEADER
I20250904 07:54:43.475430 11025 ts_tablet_manager.cc:1428] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250904 07:54:43.475513 11027 consensus_queue.cc:237] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } }
I20250904 07:54:43.477072 10870 catalog_manager.cc:5582] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ce532035e8e4611a15e5e39098b9006 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9ce532035e8e4611a15e5e39098b9006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce532035e8e4611a15e5e39098b9006" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40613 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:43.515022 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:43.523988 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:43.524305 413 tablet_replica.cc:331] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006: stopping tablet replica
I20250904 07:54:43.524504 413 raft_consensus.cc:2241] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.524685 413 raft_consensus.cc:2270] T 1c725f572ed24ed18c65c57326a4cb3c P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.525599 413 tablet_replica.cc:331] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006: stopping tablet replica
I20250904 07:54:43.525730 413 raft_consensus.cc:2241] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.525866 413 raft_consensus.cc:2270] T 193a9d1de28a4dbd95a2e235a04ac533 P 9ce532035e8e4611a15e5e39098b9006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.530638 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:43.543643 413 master.cc:561] Master@127.0.103.126:37081 shutting down...
I20250904 07:54:43.553305 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.553493 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.553615 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1a93f862923443648a47d479f02438b3: stopping tablet replica
I20250904 07:54:43.557667 413 master.cc:583] Master@127.0.103.126:37081 shutdown complete.
[ OK ] Params/ClientTestImmutableColumn.TestUpdateIgnore/2 (253 ms)
[ RUN ] Params/ClientTestImmutableColumn.TestUpsert/2
I20250904 07:54:43.569739 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:33453
I20250904 07:54:43.570291 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:43.572515 11037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.572649 11038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.573983 11040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:43.574275 413 server_base.cc:1047] running on GCE node
I20250904 07:54:43.574486 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:43.574533 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:43.574555 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972483574552 us; error 0 us; skew 500 ppm
I20250904 07:54:43.574694 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:43.575810 413 webserver.cc:480] Webserver started at http://127.0.103.126:34751/ using document root <none> and password file <none>
I20250904 07:54:43.575996 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:43.576058 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:43.576167 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:43.576674 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/master-0-root/instance:
uuid: "3ee00662b89a4688a34c83a746efaf2d"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.578864 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250904 07:54:43.580197 11045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.580546 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250904 07:54:43.580634 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/master-0-root
uuid: "3ee00662b89a4688a34c83a746efaf2d"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.580746 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:43.591681 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:43.592068 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:43.607282 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:33453
I20250904 07:54:43.607297 11106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:33453 every 8 connection(s)
I20250904 07:54:43.608570 11107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.612954 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d: Bootstrap starting.
I20250904 07:54:43.614061 11107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.615406 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d: No bootstrap required, opened a new log
I20250904 07:54:43.615934 11107 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER }
I20250904 07:54:43.616048 11107 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.616098 11107 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee00662b89a4688a34c83a746efaf2d, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.616286 11107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER }
I20250904 07:54:43.616384 11107 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.616441 11107 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.616510 11107 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.618364 11107 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER }
I20250904 07:54:43.618522 11107 leader_election.cc:304] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ee00662b89a4688a34c83a746efaf2d; no voters:
I20250904 07:54:43.618929 11107 leader_election.cc:290] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.619036 11110 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.619887 11107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [sys.catalog]: configured and running, proceeding with master startup.
I20250904 07:54:43.621031 11110 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 1 LEADER]: Becoming Leader. State: Replica: 3ee00662b89a4688a34c83a746efaf2d, State: Running, Role: LEADER
I20250904 07:54:43.621311 11110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER }
I20250904 07:54:43.622222 11111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ee00662b89a4688a34c83a746efaf2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER } }
I20250904 07:54:43.622360 11111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:43.622467 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ee00662b89a4688a34c83a746efaf2d. Latest consensus state: current_term: 1 leader_uuid: "3ee00662b89a4688a34c83a746efaf2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee00662b89a4688a34c83a746efaf2d" member_type: VOTER } }
I20250904 07:54:43.622637 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [sys.catalog]: This master's current role is: LEADER
I20250904 07:54:43.633493 11116 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 07:54:43.635263 11116 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 07:54:43.636190 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250904 07:54:43.638415 11116 catalog_manager.cc:1349] Generated new cluster ID: 5e8ca13018ea4d819197a2ac69b8b603
I20250904 07:54:43.638494 11116 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 07:54:43.673527 11116 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 07:54:43.674566 11116 catalog_manager.cc:1506] Loading token signing keys...
I20250904 07:54:43.684334 11116 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d: Generated new TSK 0
I20250904 07:54:43.684558 11116 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 07:54:43.700788 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 07:54:43.703186 11128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.713480 11129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 07:54:43.713882 11131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 07:54:43.714468 413 server_base.cc:1047] running on GCE node
I20250904 07:54:43.714685 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 07:54:43.714740 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 07:54:43.714783 413 hybrid_clock.cc:648] HybridClock initialized: now 1756972483714780 us; error 0 us; skew 500 ppm
I20250904 07:54:43.714923 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 07:54:43.716203 413 webserver.cc:480] Webserver started at http://127.0.103.65:41081/ using document root <none> and password file <none>
I20250904 07:54:43.716404 413 fs_manager.cc:362] Metadata directory not provided
I20250904 07:54:43.716468 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 07:54:43.716588 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 07:54:43.717239 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/ts-0-root/instance:
uuid: "eec9aba888664b2eba4ef65eed0829f3"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.719555 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250904 07:54:43.721076 11136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.721410 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250904 07:54:43.721495 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/ts-0-root
uuid: "eec9aba888664b2eba4ef65eed0829f3"
format_stamp: "Formatted at 2025-09-04 07:54:43 on dist-test-slave-ghlz"
I20250904 07:54:43.721601 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3MnXLe/test-tmp/client-test.6.Params_ClientTestImmutableColumn.TestUpsert_2.1756972373064810-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 07:54:43.729784 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 07:54:43.730177 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 07:54:43.732292 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 07:54:43.732364 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.732451 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 07:54:43.732497 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 07:54:43.756757 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35873
I20250904 07:54:43.756786 11206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:35873 every 8 connection(s)
I20250904 07:54:43.758194 11207 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33453
I20250904 07:54:43.758303 11207 heartbeater.cc:461] Registering TS with master...
I20250904 07:54:43.758562 11207 heartbeater.cc:507] Master 127.0.103.126:33453 requested a full tablet report, sending...
I20250904 07:54:43.759308 11062 ts_manager.cc:194] Registered new tserver with Master: eec9aba888664b2eba4ef65eed0829f3 (127.0.103.65:35873)
I20250904 07:54:43.759924 413 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002249417s
I20250904 07:54:43.760668 11062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47278
I20250904 07:54:43.770015 11062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47306:
name: "client-testtb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "non_null_with_default"
type: INT32
is_key: false
is_nullable: false
read_default_value: "90\000\000"
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "imm_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: true
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\t\000\000\000""\004\001\000\t\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 07:54:43.772135 11207 heartbeater.cc:499] Master 127.0.103.126:33453 was elected leader, sending a full tablet report...
I20250904 07:54:43.783507 11167 tablet_service.cc:1468] Processing CreateTablet for tablet 4eaa282bea024f9db0d72c15013e223e (DEFAULT_TABLE table=client-testtb [id=cd8341505f6a405590ee2a3bcd6951d2]), partition=RANGE (key) PARTITION VALUES < 9
I20250904 07:54:43.783628 11166 tablet_service.cc:1468] Processing CreateTablet for tablet e1bb2938c9b84f0c8cd142bef63d023b (DEFAULT_TABLE table=client-testtb [id=cd8341505f6a405590ee2a3bcd6951d2]), partition=RANGE (key) PARTITION 9 <= VALUES
I20250904 07:54:43.783982 11167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eaa282bea024f9db0d72c15013e223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.784194 11166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1bb2938c9b84f0c8cd142bef63d023b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 07:54:43.789214 11217 tablet_bootstrap.cc:492] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: Bootstrap starting.
I20250904 07:54:43.790416 11217 tablet_bootstrap.cc:654] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.791932 11217 tablet_bootstrap.cc:492] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: No bootstrap required, opened a new log
I20250904 07:54:43.792049 11217 ts_tablet_manager.cc:1397] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250904 07:54:43.792532 11217 raft_consensus.cc:357] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.792647 11217 raft_consensus.cc:383] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.792726 11217 raft_consensus.cc:738] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec9aba888664b2eba4ef65eed0829f3, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.792937 11217 consensus_queue.cc:260] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.793067 11217 raft_consensus.cc:397] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.793128 11217 raft_consensus.cc:491] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.793200 11217 raft_consensus.cc:3058] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.794888 11217 raft_consensus.cc:513] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.795089 11217 leader_election.cc:304] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec9aba888664b2eba4ef65eed0829f3; no voters:
I20250904 07:54:43.799595 11217 leader_election.cc:290] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.799760 11219 raft_consensus.cc:2802] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.800812 11217 ts_tablet_manager.cc:1428] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20250904 07:54:43.801067 11217 tablet_bootstrap.cc:492] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: Bootstrap starting.
I20250904 07:54:43.801836 11219 raft_consensus.cc:695] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 1 LEADER]: Becoming Leader. State: Replica: eec9aba888664b2eba4ef65eed0829f3, State: Running, Role: LEADER
I20250904 07:54:43.802130 11219 consensus_queue.cc:237] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.803000 11217 tablet_bootstrap.cc:654] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: Neither blocks nor log segments found. Creating new log.
I20250904 07:54:43.804208 11062 catalog_manager.cc:5582] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eec9aba888664b2eba4ef65eed0829f3 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "eec9aba888664b2eba4ef65eed0829f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:43.807616 11217 tablet_bootstrap.cc:492] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: No bootstrap required, opened a new log
I20250904 07:54:43.807752 11217 ts_tablet_manager.cc:1397] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250904 07:54:43.808488 11217 raft_consensus.cc:357] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.808642 11217 raft_consensus.cc:383] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 07:54:43.808705 11217 raft_consensus.cc:738] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec9aba888664b2eba4ef65eed0829f3, State: Initialized, Role: FOLLOWER
I20250904 07:54:43.808976 11217 consensus_queue.cc:260] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.809139 11217 raft_consensus.cc:397] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 07:54:43.809226 11217 raft_consensus.cc:491] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 07:54:43.809320 11217 raft_consensus.cc:3058] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 07:54:43.811427 11217 raft_consensus.cc:513] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.811591 11217 leader_election.cc:304] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eec9aba888664b2eba4ef65eed0829f3; no voters:
I20250904 07:54:43.811751 11217 leader_election.cc:290] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 07:54:43.811823 11221 raft_consensus.cc:2802] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 07:54:43.811988 11221 raft_consensus.cc:695] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 1 LEADER]: Becoming Leader. State: Replica: eec9aba888664b2eba4ef65eed0829f3, State: Running, Role: LEADER
I20250904 07:54:43.812106 11217 ts_tablet_manager.cc:1428] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250904 07:54:43.812180 11221 consensus_queue.cc:237] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } }
I20250904 07:54:43.814201 11061 catalog_manager.cc:5582] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eec9aba888664b2eba4ef65eed0829f3 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "eec9aba888664b2eba4ef65eed0829f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eec9aba888664b2eba4ef65eed0829f3" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35873 } health_report { overall_health: HEALTHY } } }
I20250904 07:54:43.849718 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250904 07:54:43.859817 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250904 07:54:43.860077 413 tablet_replica.cc:331] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3: stopping tablet replica
I20250904 07:54:43.860327 413 raft_consensus.cc:2241] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.860495 413 raft_consensus.cc:2270] T e1bb2938c9b84f0c8cd142bef63d023b P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.861336 413 tablet_replica.cc:331] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3: stopping tablet replica
I20250904 07:54:43.861438 413 raft_consensus.cc:2241] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.861605 413 raft_consensus.cc:2270] T 4eaa282bea024f9db0d72c15013e223e P eec9aba888664b2eba4ef65eed0829f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.866348 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250904 07:54:43.870687 413 master.cc:561] Master@127.0.103.126:33453 shutting down...
I20250904 07:54:43.882332 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 1 LEADER]: Raft consensus shutting down.
I20250904 07:54:43.882550 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 07:54:43.882738 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3ee00662b89a4688a34c83a746efaf2d: stopping tablet replica
I20250904 07:54:43.897154 413 master.cc:583] Master@127.0.103.126:33453 shutdown complete.
[ OK ] Params/ClientTestImmutableColumn.TestUpsert/2 (359 ms)
[----------] 2 tests from Params/ClientTestImmutableColumn (612 ms total)
[----------] Global test environment tear-down
[==========] 28 tests from 8 test suites ran. (110792 ms total)
[ PASSED ] 27 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] MultiTServerClientTest.WriteWhileRestartingMultipleTabletServers
1 FAILED TEST
I20250904 07:54:43.930403 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/master_proxy_rpc.cc:203: suppressed but not reported on 5 messages since previous log ~2 seconds ago
I20250904 07:54:43.930507 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 59 messages since previous log ~267 seconds ago
I20250904 07:54:43.930581 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 14 messages since previous log ~29 seconds ago
I20250904 07:54:43.930622 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4671: suppressed but not reported on 2 messages since previous log ~61 seconds ago