[==========] Running 25 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 21 tests from TxnParticipantITest
[ RUN ] TxnParticipantITest.TestReplicateParticipantOps
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 03:05:56.722965 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:38771
I20250113 03:05:56.723258 20553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 03:05:56.723388 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:05:56.727586 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.727746 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:05:56.728515 20553 server_base.cc:1034] running on GCE node
W20250113 03:05:56.728994 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:05:56.729195 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:05:56.729238 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:05:56.729255 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737556729256 us; error 0 us; skew 500 ppm
I20250113 03:05:56.730202 20553 webserver.cc:458] Webserver started at http://127.20.18.126:39491/ using document root <none> and password file <none>
I20250113 03:05:56.730324 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:05:56.730363 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:05:56.730414 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:05:56.731693 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "870ae9443c8d43e3921b9f0ef3710d8a"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.732723 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250113 03:05:56.733274 20566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.733456 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250113 03:05:56.733515 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "870ae9443c8d43e3921b9f0ef3710d8a"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.733556 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:05:56.746572 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:05:56.746767 20553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 03:05:56.746841 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:05:56.750108 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:38771
I20250113 03:05:56.750139 20618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:38771 every 8 connection(s)
I20250113 03:05:56.750613 20619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:05:56.752902 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: Bootstrap starting.
I20250113 03:05:56.753703 20619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: Neither blocks nor log segments found. Creating new log.
I20250113 03:05:56.753911 20619 log.cc:826] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: Log is configured to *not* fsync() on all Append() calls
I20250113 03:05:56.754381 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: No bootstrap required, opened a new log
I20250113 03:05:56.755553 20619 raft_consensus.cc:357] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER }
I20250113 03:05:56.755630 20619 raft_consensus.cc:383] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:05:56.755656 20619 raft_consensus.cc:738] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870ae9443c8d43e3921b9f0ef3710d8a, State: Initialized, Role: FOLLOWER
I20250113 03:05:56.755759 20619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER }
I20250113 03:05:56.755803 20619 raft_consensus.cc:397] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:05:56.755838 20619 raft_consensus.cc:491] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:05:56.755873 20619 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:05:56.756278 20619 raft_consensus.cc:513] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER }
I20250113 03:05:56.756356 20619 leader_election.cc:304] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 870ae9443c8d43e3921b9f0ef3710d8a; no voters:
I20250113 03:05:56.756471 20619 leader_election.cc:290] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:05:56.756505 20622 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:05:56.756654 20622 raft_consensus.cc:695] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 1 LEADER]: Becoming Leader. State: Replica: 870ae9443c8d43e3921b9f0ef3710d8a, State: Running, Role: LEADER
I20250113 03:05:56.756669 20619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:05:56.756762 20622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER }
I20250113 03:05:56.757045 20624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870ae9443c8d43e3921b9f0ef3710d8a. Latest consensus state: current_term: 1 leader_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER } }
I20250113 03:05:56.757190 20624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [sys.catalog]: This master's current role is: LEADER
I20250113 03:05:56.757041 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "870ae9443c8d43e3921b9f0ef3710d8a" member_type: VOTER } }
I20250113 03:05:56.757280 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [sys.catalog]: This master's current role is: LEADER
I20250113 03:05:56.757550 20630 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:05:56.758028 20630 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:05:56.758137 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:05:56.759277 20630 catalog_manager.cc:1348] Generated new cluster ID: f9dc7f3da29b4c23b41d21ce0041a166
I20250113 03:05:56.759338 20630 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:05:56.770006 20630 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:05:56.770442 20630 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:05:56.776147 20630 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: Generated new TSK 0
I20250113 03:05:56.776273 20630 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:05:56.789952 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:05:56.791323 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.791950 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.792701 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:05:56.793387 20553 server_base.cc:1034] running on GCE node
I20250113 03:05:56.793509 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:05:56.793535 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:05:56.793555 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737556793554 us; error 0 us; skew 500 ppm
I20250113 03:05:56.794215 20553 webserver.cc:458] Webserver started at http://127.20.18.65:40577/ using document root <none> and password file <none>
I20250113 03:05:56.794296 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:05:56.794337 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:05:56.794380 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:05:56.794632 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "711a328b88d94f64aba7b7cc0c806f4a"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.795403 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:05:56.795830 20649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.795954 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.795995 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "711a328b88d94f64aba7b7cc0c806f4a"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.796036 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:05:56.815708 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:05:56.815872 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:05:56.816033 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:05:56.816310 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:05:56.816340 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.816359 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:05:56.816382 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.820112 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:38357
I20250113 03:05:56.820147 20712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:38357 every 8 connection(s)
I20250113 03:05:56.820470 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:05:56.821717 20718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.822219 20720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.822985 20717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:05:56.823638 20553 server_base.cc:1034] running on GCE node
I20250113 03:05:56.823747 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:05:56.823771 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:05:56.823791 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737556823790 us; error 0 us; skew 500 ppm
I20250113 03:05:56.824405 20553 webserver.cc:458] Webserver started at http://127.20.18.66:44193/ using document root <none> and password file <none>
I20250113 03:05:56.824491 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:05:56.824546 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:05:56.824589 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:05:56.824810 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "870bd3260fe041818bb4e08da3821d63"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.825608 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:05:56.826028 20725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.826141 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.826185 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "870bd3260fe041818bb4e08da3821d63"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.826217 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:05:56.832955 20713 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38771
I20250113 03:05:56.833036 20713 heartbeater.cc:463] Registering TS with master...
I20250113 03:05:56.833173 20713 heartbeater.cc:510] Master 127.20.18.126:38771 requested a full tablet report, sending...
I20250113 03:05:56.833633 20583 ts_manager.cc:194] Registered new tserver with Master: 711a328b88d94f64aba7b7cc0c806f4a (127.20.18.65:38357)
I20250113 03:05:56.834465 20583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49114
I20250113 03:05:56.836194 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:05:56.836442 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:05:56.836608 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:05:56.836819 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:05:56.836851 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.836885 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:05:56.836911 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.840713 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:39805
I20250113 03:05:56.840781 20788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:39805 every 8 connection(s)
I20250113 03:05:56.840998 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:05:56.842192 20791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.842844 20794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:05:56.843541 20792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:05:56.844295 20553 server_base.cc:1034] running on GCE node
I20250113 03:05:56.844378 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:05:56.844406 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:05:56.844424 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737556844424 us; error 0 us; skew 500 ppm
I20250113 03:05:56.844854 20789 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38771
I20250113 03:05:56.844951 20789 heartbeater.cc:463] Registering TS with master...
I20250113 03:05:56.845062 20789 heartbeater.cc:510] Master 127.20.18.126:38771 requested a full tablet report, sending...
I20250113 03:05:56.845127 20553 webserver.cc:458] Webserver started at http://127.20.18.67:45233/ using document root <none> and password file <none>
I20250113 03:05:56.845225 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:05:56.845264 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:05:56.845310 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:05:56.845310 20583 ts_manager.cc:194] Registered new tserver with Master: 870bd3260fe041818bb4e08da3821d63 (127.20.18.66:39805)
I20250113 03:05:56.845593 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "159f8f96d0004a67a035d6774128f267"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.845773 20583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49122
I20250113 03:05:56.846410 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:05:56.846807 20800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.846897 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.846936 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "159f8f96d0004a67a035d6774128f267"
format_stamp: "Formatted at 2025-01-13 03:05:56 on dist-test-slave-d4dl"
I20250113 03:05:56.846973 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:05:56.873327 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:05:56.873531 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:05:56.873660 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:05:56.873837 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:05:56.873871 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.873891 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:05:56.873903 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.877913 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:43975
I20250113 03:05:56.878015 20863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:43975 every 8 connection(s)
I20250113 03:05:56.880874 20864 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38771
I20250113 03:05:56.880929 20864 heartbeater.cc:463] Registering TS with master...
I20250113 03:05:56.881031 20864 heartbeater.cc:510] Master 127.20.18.126:38771 requested a full tablet report, sending...
I20250113 03:05:56.881278 20583 ts_manager.cc:194] Registered new tserver with Master: 159f8f96d0004a67a035d6774128f267 (127.20.18.67:43975)
I20250113 03:05:56.881402 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003311005s
I20250113 03:05:56.881719 20583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49126
I20250113 03:05:56.881801 20553 test_util.cc:274] Using random seed: -1188732538
I20250113 03:05:56.887991 20583 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49142:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:05:56.888206 20583 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:05:56.892829 20753 tablet_service.cc:1467] Processing CreateTablet for tablet 6b658e715525495fb2e871a196d9a0a7 (DEFAULT_TABLE table=test-workload [id=6a76198c196641118fd97c4ef9668143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:05:56.892967 20753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b658e715525495fb2e871a196d9a0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:05:56.893172 20828 tablet_service.cc:1467] Processing CreateTablet for tablet 6b658e715525495fb2e871a196d9a0a7 (DEFAULT_TABLE table=test-workload [id=6a76198c196641118fd97c4ef9668143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:05:56.893272 20828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b658e715525495fb2e871a196d9a0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:05:56.894033 20677 tablet_service.cc:1467] Processing CreateTablet for tablet 6b658e715525495fb2e871a196d9a0a7 (DEFAULT_TABLE table=test-workload [id=6a76198c196641118fd97c4ef9668143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:05:56.894133 20677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b658e715525495fb2e871a196d9a0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:05:56.894258 20879 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: Bootstrap starting.
I20250113 03:05:56.894479 20880 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: Bootstrap starting.
I20250113 03:05:56.894623 20879 tablet_bootstrap.cc:654] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: Neither blocks nor log segments found. Creating new log.
I20250113 03:05:56.894706 20880 tablet_bootstrap.cc:654] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: Neither blocks nor log segments found. Creating new log.
I20250113 03:05:56.895112 20880 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: No bootstrap required, opened a new log
I20250113 03:05:56.895190 20879 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: No bootstrap required, opened a new log
I20250113 03:05:56.895236 20879 ts_tablet_manager.cc:1397] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:05:56.895243 20883 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: Bootstrap starting.
I20250113 03:05:56.895339 20879 raft_consensus.cc:357] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.895421 20879 raft_consensus.cc:383] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:05:56.895454 20879 raft_consensus.cc:738] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f8f96d0004a67a035d6774128f267, State: Initialized, Role: FOLLOWER
I20250113 03:05:56.895522 20883 tablet_bootstrap.cc:654] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: Neither blocks nor log segments found. Creating new log.
I20250113 03:05:56.895185 20880 ts_tablet_manager.cc:1397] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:05:56.895506 20879 consensus_queue.cc:260] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.895681 20880 raft_consensus.cc:357] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.895720 20879 ts_tablet_manager.cc:1428] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:05:56.895781 20864 heartbeater.cc:502] Master 127.20.18.126:38771 was elected leader, sending a full tablet report...
I20250113 03:05:56.895849 20880 raft_consensus.cc:383] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:05:56.896019 20883 tablet_bootstrap.cc:492] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: No bootstrap required, opened a new log
I20250113 03:05:56.896024 20880 raft_consensus.cc:738] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870bd3260fe041818bb4e08da3821d63, State: Initialized, Role: FOLLOWER
I20250113 03:05:56.896102 20883 ts_tablet_manager.cc:1397] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:05:56.896136 20880 consensus_queue.cc:260] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.896450 20883 raft_consensus.cc:357] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.896549 20883 raft_consensus.cc:383] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:05:56.896644 20880 ts_tablet_manager.cc:1428] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:05:56.896669 20789 heartbeater.cc:502] Master 127.20.18.126:38771 was elected leader, sending a full tablet report...
I20250113 03:05:56.896692 20883 raft_consensus.cc:738] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711a328b88d94f64aba7b7cc0c806f4a, State: Initialized, Role: FOLLOWER
I20250113 03:05:56.896783 20883 consensus_queue.cc:260] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:56.896919 20883 ts_tablet_manager.cc:1428] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:05:56.896955 20713 heartbeater.cc:502] Master 127.20.18.126:38771 was elected leader, sending a full tablet report...
W20250113 03:05:57.070528 20714 tablet.cc:2367] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 03:05:57.148128 20887 raft_consensus.cc:491] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:05:57.148219 20887 raft_consensus.cc:513] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:57.148447 20887 leader_election.cc:290] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159f8f96d0004a67a035d6774128f267 (127.20.18.67:43975), 870bd3260fe041818bb4e08da3821d63 (127.20.18.66:39805)
I20250113 03:05:57.151332 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b658e715525495fb2e871a196d9a0a7" candidate_uuid: "711a328b88d94f64aba7b7cc0c806f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159f8f96d0004a67a035d6774128f267" is_pre_election: true
I20250113 03:05:57.151335 20763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b658e715525495fb2e871a196d9a0a7" candidate_uuid: "711a328b88d94f64aba7b7cc0c806f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870bd3260fe041818bb4e08da3821d63" is_pre_election: true
I20250113 03:05:57.151444 20838 raft_consensus.cc:2463] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 711a328b88d94f64aba7b7cc0c806f4a in term 0.
I20250113 03:05:57.151450 20763 raft_consensus.cc:2463] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 711a328b88d94f64aba7b7cc0c806f4a in term 0.
I20250113 03:05:57.151871 20652 leader_election.cc:304] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159f8f96d0004a67a035d6774128f267, 711a328b88d94f64aba7b7cc0c806f4a; no voters:
I20250113 03:05:57.151968 20887 raft_consensus.cc:2798] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:05:57.152010 20887 raft_consensus.cc:491] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:05:57.152024 20887 raft_consensus.cc:3054] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:05:57.152514 20887 raft_consensus.cc:513] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:57.152633 20887 leader_election.cc:290] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [CANDIDATE]: Term 1 election: Requested vote from peers 159f8f96d0004a67a035d6774128f267 (127.20.18.67:43975), 870bd3260fe041818bb4e08da3821d63 (127.20.18.66:39805)
I20250113 03:05:57.152771 20763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b658e715525495fb2e871a196d9a0a7" candidate_uuid: "711a328b88d94f64aba7b7cc0c806f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870bd3260fe041818bb4e08da3821d63"
I20250113 03:05:57.152803 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b658e715525495fb2e871a196d9a0a7" candidate_uuid: "711a328b88d94f64aba7b7cc0c806f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159f8f96d0004a67a035d6774128f267"
I20250113 03:05:57.152823 20763 raft_consensus.cc:3054] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:05:57.152855 20838 raft_consensus.cc:3054] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:05:57.153395 20763 raft_consensus.cc:2463] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711a328b88d94f64aba7b7cc0c806f4a in term 1.
I20250113 03:05:57.153402 20838 raft_consensus.cc:2463] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 711a328b88d94f64aba7b7cc0c806f4a in term 1.
I20250113 03:05:57.153554 20650 leader_election.cc:304] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 711a328b88d94f64aba7b7cc0c806f4a, 870bd3260fe041818bb4e08da3821d63; no voters:
I20250113 03:05:57.153635 20887 raft_consensus.cc:2798] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:05:57.153774 20887 raft_consensus.cc:695] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 1 LEADER]: Becoming Leader. State: Replica: 711a328b88d94f64aba7b7cc0c806f4a, State: Running, Role: LEADER
I20250113 03:05:57.153831 20887 consensus_queue.cc:237] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } }
I20250113 03:05:57.154434 20582 catalog_manager.cc:5526] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 711a328b88d94f64aba7b7cc0c806f4a (127.20.18.65). New cstate: current_term: 1 leader_uuid: "711a328b88d94f64aba7b7cc0c806f4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "711a328b88d94f64aba7b7cc0c806f4a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38357 } health_report { overall_health: HEALTHY } } }
I20250113 03:05:57.180653 20838 raft_consensus.cc:1270] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 1 FOLLOWER]: Refusing update from remote peer 711a328b88d94f64aba7b7cc0c806f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:05:57.180655 20763 raft_consensus.cc:1270] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 1 FOLLOWER]: Refusing update from remote peer 711a328b88d94f64aba7b7cc0c806f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:05:57.180881 20889 consensus_queue.cc:1035] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "159f8f96d0004a67a035d6774128f267" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:05:57.180969 20887 consensus_queue.cc:1035] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "870bd3260fe041818bb4e08da3821d63" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:05:57.182219 20896 mvcc.cc:204] Tried to move back new op lower bound from 7113677034210459648 to 7113677034102620160. Current Snapshot: MvccSnapshot[applied={T|T < 7113677034210459648}]
I20250113 03:06:07.181013 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:06:07.184484 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:07.184594 20553 tablet_replica.cc:331] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a: stopping tablet replica
I20250113 03:06:07.184664 20553 raft_consensus.cc:2238] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:07.184737 20553 raft_consensus.cc:2267] T 6b658e715525495fb2e871a196d9a0a7 P 711a328b88d94f64aba7b7cc0c806f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:07.202064 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:06:07.202966 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:06:07.205521 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:07.205629 20553 tablet_replica.cc:331] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63: stopping tablet replica
I20250113 03:06:07.205670 20553 raft_consensus.cc:2238] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:07.205714 20553 raft_consensus.cc:2267] T 6b658e715525495fb2e871a196d9a0a7 P 870bd3260fe041818bb4e08da3821d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:07.218672 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:06:07.219251 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:06:07.221395 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:07.221490 20553 tablet_replica.cc:331] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267: stopping tablet replica
I20250113 03:06:07.221534 20553 raft_consensus.cc:2238] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:07.221572 20553 raft_consensus.cc:2267] T 6b658e715525495fb2e871a196d9a0a7 P 159f8f96d0004a67a035d6774128f267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:07.243207 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:06:07.243973 20553 master.cc:537] Master@127.20.18.126:38771 shutting down...
I20250113 03:06:07.246037 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:07.246101 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:07.246132 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 870ae9443c8d43e3921b9f0ef3710d8a: stopping tablet replica
I20250113 03:06:07.257443 20553 master.cc:559] Master@127.20.18.126:38771 shutdown complete.
[ OK ] TxnParticipantITest.TestReplicateParticipantOps (10552 ms)
[ RUN ] TxnParticipantITest.TestWaitOnFinalizeCommit
I20250113 03:06:07.275787 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:38395
I20250113 03:06:07.275928 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:07.277848 20961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.278067 20958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.278787 20959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:07.279428 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:07.279510 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:07.279538 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:07.279557 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737567279558 us; error 0 us; skew 500 ppm
I20250113 03:06:07.280133 20553 webserver.cc:458] Webserver started at http://127.20.18.126:46103/ using document root <none> and password file <none>
I20250113 03:06:07.280206 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:07.280251 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:07.280292 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:07.280587 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "6049074107164e38ac60c2b62d5bf051"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.281433 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:07.281860 20966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.281996 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.282035 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "6049074107164e38ac60c2b62d5bf051"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.282081 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:07.296128 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:07.296284 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:07.298902 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:38395
I20250113 03:06:07.300968 21018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:38395 every 8 connection(s)
I20250113 03:06:07.301148 21019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:07.303246 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051: Bootstrap starting.
I20250113 03:06:07.303560 21019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:07.303994 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051: No bootstrap required, opened a new log
I20250113 03:06:07.304111 21019 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER }
I20250113 03:06:07.304155 21019 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:07.304169 21019 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6049074107164e38ac60c2b62d5bf051, State: Initialized, Role: FOLLOWER
I20250113 03:06:07.304215 21019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER }
I20250113 03:06:07.304252 21019 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:06:07.304282 21019 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:06:07.304308 21019 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:07.304818 21019 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER }
I20250113 03:06:07.304867 21019 leader_election.cc:304] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6049074107164e38ac60c2b62d5bf051; no voters:
I20250113 03:06:07.304960 21019 leader_election.cc:290] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:06:07.304999 21022 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:07.305086 21019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:06:07.305114 21022 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 1 LEADER]: Becoming Leader. State: Replica: 6049074107164e38ac60c2b62d5bf051, State: Running, Role: LEADER
I20250113 03:06:07.305156 21022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER }
I20250113 03:06:07.305311 21022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6049074107164e38ac60c2b62d5bf051. Latest consensus state: current_term: 1 leader_uuid: "6049074107164e38ac60c2b62d5bf051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER } }
I20250113 03:06:07.305322 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6049074107164e38ac60c2b62d5bf051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6049074107164e38ac60c2b62d5bf051" member_type: VOTER } }
I20250113 03:06:07.305367 21022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:07.305373 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:07.305732 21026 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:06:07.305873 21026 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:06:07.306315 21026 catalog_manager.cc:1348] Generated new cluster ID: 7dff6332fcc84748917408f5c4dfd9e7
I20250113 03:06:07.306336 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:06:07.306358 21026 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:06:07.317691 21026 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:06:07.318033 21026 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:06:07.322772 21026 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051: Generated new TSK 0
I20250113 03:06:07.322834 21026 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:06:07.337946 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:07.339211 21040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.339928 21043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:07.340649 20553 server_base.cc:1034] running on GCE node
W20250113 03:06:07.341252 21041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:07.341390 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:07.341420 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:07.341439 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737567341438 us; error 0 us; skew 500 ppm
I20250113 03:06:07.341993 20553 webserver.cc:458] Webserver started at http://127.20.18.65:39459/ using document root <none> and password file <none>
I20250113 03:06:07.342068 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:07.342106 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:07.342149 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:07.342384 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "384c4085d2224cab901f93c15547a9ac"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.343081 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:07.343437 21048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.343537 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.343572 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "384c4085d2224cab901f93c15547a9ac"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.343612 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:07.356609 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:07.356750 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:07.356866 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:07.357014 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:07.357046 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.357072 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:07.357113 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.359412 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:36177
I20250113 03:06:07.359596 21111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:36177 every 8 connection(s)
I20250113 03:06:07.359751 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:07.360862 21116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.361614 21117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.362308 21119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:07.362713 21112 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38395
I20250113 03:06:07.362771 21112 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:07.362856 21112 heartbeater.cc:510] Master 127.20.18.126:38395 requested a full tablet report, sending...
I20250113 03:06:07.362941 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:07.363049 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:07.363086 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:07.363106 20983 ts_manager.cc:194] Registered new tserver with Master: 384c4085d2224cab901f93c15547a9ac (127.20.18.65:36177)
I20250113 03:06:07.363117 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737567363116 us; error 0 us; skew 500 ppm
I20250113 03:06:07.363590 20983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35630
I20250113 03:06:07.363765 20553 webserver.cc:458] Webserver started at http://127.20.18.66:40833/ using document root <none> and password file <none>
I20250113 03:06:07.363848 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:07.363890 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:07.363932 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:07.364172 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "699d54c64f874b67b61e5439eba8a218"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.364951 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:07.365343 21124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.365455 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.365499 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "699d54c64f874b67b61e5439eba8a218"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.365535 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:07.373056 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:07.373183 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:07.373275 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:07.373402 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:07.373422 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.373446 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:07.373461 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.375785 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:45445
I20250113 03:06:07.375836 21187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:45445 every 8 connection(s)
I20250113 03:06:07.376060 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:07.377188 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.377904 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:07.378587 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:07.378683 21188 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38395
I20250113 03:06:07.378738 21188 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:07.378820 21188 heartbeater.cc:510] Master 127.20.18.126:38395 requested a full tablet report, sending...
I20250113 03:06:07.379011 20983 ts_manager.cc:194] Registered new tserver with Master: 699d54c64f874b67b61e5439eba8a218 (127.20.18.66:45445)
I20250113 03:06:07.379215 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:07.379321 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:07.379354 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:07.379374 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737567379373 us; error 0 us; skew 500 ppm
I20250113 03:06:07.379383 20983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35646
I20250113 03:06:07.379923 20553 webserver.cc:458] Webserver started at http://127.20.18.67:42087/ using document root <none> and password file <none>
I20250113 03:06:07.379993 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:07.380034 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:07.380075 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:07.380292 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "a8f09351821c4c619da9feb0f2522fe3"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.381057 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:07.381439 21199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.381541 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.381579 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "a8f09351821c4c619da9feb0f2522fe3"
format_stamp: "Formatted at 2025-01-13 03:06:07 on dist-test-slave-d4dl"
I20250113 03:06:07.381620 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:07.391552 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:07.391729 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:07.391872 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:07.392024 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:07.392047 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.392072 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:07.392094 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.395004 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:40269
I20250113 03:06:07.395046 21262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:40269 every 8 connection(s)
I20250113 03:06:07.397521 21263 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38395
I20250113 03:06:07.397565 21263 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:07.397643 21263 heartbeater.cc:510] Master 127.20.18.126:38395 requested a full tablet report, sending...
I20250113 03:06:07.397873 20983 ts_manager.cc:194] Registered new tserver with Master: a8f09351821c4c619da9feb0f2522fe3 (127.20.18.67:40269)
I20250113 03:06:07.398167 20983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35652
I20250113 03:06:07.398444 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003295234s
I20250113 03:06:07.398748 20553 test_util.cc:274] Using random seed: -1178215588
I20250113 03:06:07.402786 20983 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35658:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:06:07.402913 20983 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:06:07.406932 21227 tablet_service.cc:1467] Processing CreateTablet for tablet 770dfd0f71354ac3b4af29f00e45b7d5 (DEFAULT_TABLE table=test-workload [id=26ceac4050384bc598cf1a8d5a61d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:07.407052 21227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770dfd0f71354ac3b4af29f00e45b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:07.407191 21076 tablet_service.cc:1467] Processing CreateTablet for tablet 770dfd0f71354ac3b4af29f00e45b7d5 (DEFAULT_TABLE table=test-workload [id=26ceac4050384bc598cf1a8d5a61d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:07.407275 21076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770dfd0f71354ac3b4af29f00e45b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:07.407620 21152 tablet_service.cc:1467] Processing CreateTablet for tablet 770dfd0f71354ac3b4af29f00e45b7d5 (DEFAULT_TABLE table=test-workload [id=26ceac4050384bc598cf1a8d5a61d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:07.407730 21152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770dfd0f71354ac3b4af29f00e45b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:07.408456 21279 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: Bootstrap starting.
I20250113 03:06:07.408763 21280 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: Bootstrap starting.
I20250113 03:06:07.408937 21279 tablet_bootstrap.cc:654] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:07.409107 21280 tablet_bootstrap.cc:654] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:07.409286 21278 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: Bootstrap starting.
I20250113 03:06:07.409528 21279 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: No bootstrap required, opened a new log
I20250113 03:06:07.409576 21279 ts_tablet_manager.cc:1397] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:07.409698 21278 tablet_bootstrap.cc:654] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:07.409729 21280 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: No bootstrap required, opened a new log
I20250113 03:06:07.409713 21279 raft_consensus.cc:357] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.409772 21280 ts_tablet_manager.cc:1397] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:07.409777 21279 raft_consensus.cc:383] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:07.409824 21279 raft_consensus.cc:738] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c4085d2224cab901f93c15547a9ac, State: Initialized, Role: FOLLOWER
I20250113 03:06:07.409890 21279 consensus_queue.cc:260] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.409916 21280 raft_consensus.cc:357] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.409970 21280 raft_consensus.cc:383] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:07.409993 21280 raft_consensus.cc:738] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699d54c64f874b67b61e5439eba8a218, State: Initialized, Role: FOLLOWER
I20250113 03:06:07.410032 21280 consensus_queue.cc:260] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.410076 21112 heartbeater.cc:502] Master 127.20.18.126:38395 was elected leader, sending a full tablet report...
I20250113 03:06:07.410171 21280 ts_tablet_manager.cc:1428] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:06:07.410256 21188 heartbeater.cc:502] Master 127.20.18.126:38395 was elected leader, sending a full tablet report...
I20250113 03:06:07.410053 21279 ts_tablet_manager.cc:1428] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:06:07.410524 21278 tablet_bootstrap.cc:492] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: No bootstrap required, opened a new log
I20250113 03:06:07.410571 21278 ts_tablet_manager.cc:1397] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:07.410687 21278 raft_consensus.cc:357] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.410727 21278 raft_consensus.cc:383] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:07.410739 21278 raft_consensus.cc:738] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f09351821c4c619da9feb0f2522fe3, State: Initialized, Role: FOLLOWER
I20250113 03:06:07.410782 21278 consensus_queue.cc:260] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.410915 21278 ts_tablet_manager.cc:1428] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:07.410945 21263 heartbeater.cc:502] Master 127.20.18.126:38395 was elected leader, sending a full tablet report...
I20250113 03:06:07.419615 21286 raft_consensus.cc:491] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:06:07.419670 21286 raft_consensus.cc:513] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.419833 21286 leader_election.cc:290] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 384c4085d2224cab901f93c15547a9ac (127.20.18.65:36177), 699d54c64f874b67b61e5439eba8a218 (127.20.18.66:45445)
I20250113 03:06:07.421743 21086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "a8f09351821c4c619da9feb0f2522fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c4085d2224cab901f93c15547a9ac" is_pre_election: true
I20250113 03:06:07.421828 21086 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f09351821c4c619da9feb0f2522fe3 in term 0.
I20250113 03:06:07.421869 21162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "a8f09351821c4c619da9feb0f2522fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d54c64f874b67b61e5439eba8a218" is_pre_election: true
I20250113 03:06:07.421916 21162 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f09351821c4c619da9feb0f2522fe3 in term 0.
I20250113 03:06:07.421958 21201 leader_election.cc:304] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384c4085d2224cab901f93c15547a9ac, a8f09351821c4c619da9feb0f2522fe3; no voters:
I20250113 03:06:07.422029 21286 raft_consensus.cc:2798] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:06:07.422062 21286 raft_consensus.cc:491] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:06:07.422078 21286 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:07.422545 21286 raft_consensus.cc:513] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.422639 21286 leader_election.cc:290] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [CANDIDATE]: Term 1 election: Requested vote from peers 384c4085d2224cab901f93c15547a9ac (127.20.18.65:36177), 699d54c64f874b67b61e5439eba8a218 (127.20.18.66:45445)
I20250113 03:06:07.422761 21162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "a8f09351821c4c619da9feb0f2522fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699d54c64f874b67b61e5439eba8a218"
I20250113 03:06:07.422808 21162 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:07.423269 21162 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f09351821c4c619da9feb0f2522fe3 in term 1.
I20250113 03:06:07.423342 21086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "a8f09351821c4c619da9feb0f2522fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c4085d2224cab901f93c15547a9ac"
I20250113 03:06:07.423389 21086 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:07.423413 21203 leader_election.cc:304] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699d54c64f874b67b61e5439eba8a218, a8f09351821c4c619da9feb0f2522fe3; no voters:
I20250113 03:06:07.423496 21286 raft_consensus.cc:2798] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:07.423619 21286 raft_consensus.cc:695] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 LEADER]: Becoming Leader. State: Replica: a8f09351821c4c619da9feb0f2522fe3, State: Running, Role: LEADER
I20250113 03:06:07.423669 21286 consensus_queue.cc:237] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:07.423993 21086 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f09351821c4c619da9feb0f2522fe3 in term 1.
I20250113 03:06:07.424142 20982 catalog_manager.cc:5526] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8f09351821c4c619da9feb0f2522fe3 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "a8f09351821c4c619da9feb0f2522fe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:07.435228 21162 raft_consensus.cc:1270] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 1 FOLLOWER]: Refusing update from remote peer a8f09351821c4c619da9feb0f2522fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:07.435237 21086 raft_consensus.cc:1270] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 1 FOLLOWER]: Refusing update from remote peer a8f09351821c4c619da9feb0f2522fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:07.435498 21288 consensus_queue.cc:1035] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:07.435585 21286 consensus_queue.cc:1035] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:17.952724 21332 raft_consensus.cc:988] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: : Instructing follower 384c4085d2224cab901f93c15547a9ac to start an election
I20250113 03:06:17.952798 21332 raft_consensus.cc:1076] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 LEADER]: Signalling peer 384c4085d2224cab901f93c15547a9ac to start an election
I20250113 03:06:17.952984 21085 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5"
dest_uuid: "384c4085d2224cab901f93c15547a9ac"
from {username='slave'} at 127.0.0.1:48236
I20250113 03:06:17.953073 21085 raft_consensus.cc:491] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:06:17.953099 21085 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:17.953668 21085 raft_consensus.cc:513] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:17.953836 21085 leader_election.cc:290] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [CANDIDATE]: Term 2 election: Requested vote from peers a8f09351821c4c619da9feb0f2522fe3 (127.20.18.67:40269), 699d54c64f874b67b61e5439eba8a218 (127.20.18.66:45445)
I20250113 03:06:17.956118 21237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "384c4085d2224cab901f93c15547a9ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 15484 } } ignore_live_leader: true dest_uuid: "a8f09351821c4c619da9feb0f2522fe3"
I20250113 03:06:17.956125 21162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770dfd0f71354ac3b4af29f00e45b7d5" candidate_uuid: "384c4085d2224cab901f93c15547a9ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 15484 } } ignore_live_leader: true dest_uuid: "699d54c64f874b67b61e5439eba8a218"
I20250113 03:06:17.956185 21237 raft_consensus.cc:3049] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:06:17.956200 21162 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:17.956207 21237 raft_consensus.cc:738] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8f09351821c4c619da9feb0f2522fe3, State: Running, Role: LEADER
I20250113 03:06:17.956259 21237 consensus_queue.cc:260] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15484, Committed index: 15484, Last appended: 1.15484, Last appended by leader: 15484, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:17.956323 21237 raft_consensus.cc:3054] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:17.956822 21162 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384c4085d2224cab901f93c15547a9ac in term 2.
I20250113 03:06:17.956972 21237 raft_consensus.cc:2463] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384c4085d2224cab901f93c15547a9ac in term 2.
I20250113 03:06:17.956982 21052 leader_election.cc:304] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384c4085d2224cab901f93c15547a9ac, 699d54c64f874b67b61e5439eba8a218; no voters:
I20250113 03:06:17.957139 21338 raft_consensus.cc:2798] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:06:17.957235 21338 raft_consensus.cc:695] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 2 LEADER]: Becoming Leader. State: Replica: 384c4085d2224cab901f93c15547a9ac, State: Running, Role: LEADER
I20250113 03:06:17.957289 21338 consensus_queue.cc:237] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15484, Committed index: 15484, Last appended: 1.15484, Last appended by leader: 15484, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } }
I20250113 03:06:17.957665 20983 catalog_manager.cc:5526] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac reported cstate change: term changed from 1 to 2, leader changed from a8f09351821c4c619da9feb0f2522fe3 (127.20.18.67) to 384c4085d2224cab901f93c15547a9ac (127.20.18.65). New cstate: current_term: 2 leader_uuid: "384c4085d2224cab901f93c15547a9ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c4085d2224cab901f93c15547a9ac" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:17.958418 21162 raft_consensus.cc:1270] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 2 FOLLOWER]: Refusing update from remote peer 384c4085d2224cab901f93c15547a9ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15484. Preceding OpId from leader: term: 2 index: 15486. (index mismatch)
I20250113 03:06:17.958457 21237 raft_consensus.cc:1270] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 2 FOLLOWER]: Refusing update from remote peer 384c4085d2224cab901f93c15547a9ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15484. Preceding OpId from leader: term: 2 index: 15486. (index mismatch)
I20250113 03:06:17.958621 21338 consensus_queue.cc:1035] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f09351821c4c619da9feb0f2522fe3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15485, Last known committed idx: 15484, Time since last communication: 0.000s
I20250113 03:06:17.958701 21339 consensus_queue.cc:1035] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "699d54c64f874b67b61e5439eba8a218" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15485, Last known committed idx: 15484, Time since last communication: 0.000s
I20250113 03:06:20.014927 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:06:20.017330 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:20.017477 20553 tablet_replica.cc:331] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac: stopping tablet replica
I20250113 03:06:20.017537 20553 raft_consensus.cc:2238] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:06:20.017592 20553 raft_consensus.cc:2267] T 770dfd0f71354ac3b4af29f00e45b7d5 P 384c4085d2224cab901f93c15547a9ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:20.033146 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:06:20.033859 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:06:20.036680 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:20.036769 20553 tablet_replica.cc:331] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218: stopping tablet replica
I20250113 03:06:20.036821 20553 raft_consensus.cc:2238] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:20.036860 20553 raft_consensus.cc:2267] T 770dfd0f71354ac3b4af29f00e45b7d5 P 699d54c64f874b67b61e5439eba8a218 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:20.049839 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:06:20.050493 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:06:20.053851 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:20.053956 20553 tablet_replica.cc:331] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3: stopping tablet replica
I20250113 03:06:20.054003 20553 raft_consensus.cc:2238] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:20.054042 20553 raft_consensus.cc:2267] T 770dfd0f71354ac3b4af29f00e45b7d5 P a8f09351821c4c619da9feb0f2522fe3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:20.076818 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:06:20.077590 20553 master.cc:537] Master@127.20.18.126:38395 shutting down...
I20250113 03:06:20.079744 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:20.079797 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:20.079826 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6049074107164e38ac60c2b62d5bf051: stopping tablet replica
I20250113 03:06:20.091236 20553 master.cc:559] Master@127.20.18.126:38395 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnFinalizeCommit (12833 ms)
[ RUN ] TxnParticipantITest.TestWaitOnAbortCommit
I20250113 03:06:20.109735 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:40137
I20250113 03:06:20.109860 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:20.111258 21378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:20.112164 20553 server_base.cc:1034] running on GCE node
W20250113 03:06:20.112876 21381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.113512 21379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:20.113626 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:20.113653 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:20.113665 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737580113665 us; error 0 us; skew 500 ppm
I20250113 03:06:20.114274 20553 webserver.cc:458] Webserver started at http://127.20.18.126:37295/ using document root <none> and password file <none>
I20250113 03:06:20.114337 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:20.114368 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:20.114395 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:20.114614 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "c9f6e89255294ffda8dfacd242493798"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.115384 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:20.115765 21386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.115885 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.115926 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "c9f6e89255294ffda8dfacd242493798"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.115974 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:20.131573 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:20.131729 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:20.134102 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:40137
I20250113 03:06:20.138283 21438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:40137 every 8 connection(s)
I20250113 03:06:20.138350 21439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:20.140630 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: Bootstrap starting.
I20250113 03:06:20.140930 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:20.141386 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: No bootstrap required, opened a new log
I20250113 03:06:20.141494 21439 raft_consensus.cc:357] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER }
I20250113 03:06:20.141539 21439 raft_consensus.cc:383] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:20.141552 21439 raft_consensus.cc:738] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f6e89255294ffda8dfacd242493798, State: Initialized, Role: FOLLOWER
I20250113 03:06:20.141600 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER }
I20250113 03:06:20.141638 21439 raft_consensus.cc:397] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:06:20.141655 21439 raft_consensus.cc:491] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:06:20.141680 21439 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:20.142117 21439 raft_consensus.cc:513] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER }
I20250113 03:06:20.142164 21439 leader_election.cc:304] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f6e89255294ffda8dfacd242493798; no voters:
I20250113 03:06:20.142256 21439 leader_election.cc:290] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:06:20.142313 21442 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:20.142388 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:06:20.142438 21442 raft_consensus.cc:695] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 1 LEADER]: Becoming Leader. State: Replica: c9f6e89255294ffda8dfacd242493798, State: Running, Role: LEADER
I20250113 03:06:20.142485 21442 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER }
I20250113 03:06:20.142671 21444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f6e89255294ffda8dfacd242493798. Latest consensus state: current_term: 1 leader_uuid: "c9f6e89255294ffda8dfacd242493798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER } }
I20250113 03:06:20.142668 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9f6e89255294ffda8dfacd242493798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f6e89255294ffda8dfacd242493798" member_type: VOTER } }
I20250113 03:06:20.142771 21444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:20.142796 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:20.143496 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20250113 03:06:20.143558 21458 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250113 03:06:20.143612 21458 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250113 03:06:20.143672 21448 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:06:20.143779 21448 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:06:20.144244 21448 catalog_manager.cc:1348] Generated new cluster ID: ba1b4fee4525418b83a58bca7f09a992
I20250113 03:06:20.144280 21448 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:06:20.149860 21448 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:06:20.150216 21448 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:06:20.163574 21448 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: Generated new TSK 0
I20250113 03:06:20.163687 21448 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:06:20.175140 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:20.176407 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.177302 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.177927 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:20.178640 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:20.178706 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:20.178728 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:20.178740 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737580178741 us; error 0 us; skew 500 ppm
I20250113 03:06:20.179328 20553 webserver.cc:458] Webserver started at http://127.20.18.65:40369/ using document root <none> and password file <none>
I20250113 03:06:20.179396 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:20.179428 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:20.179466 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:20.179679 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "77e4119e613041a1818753caed54fbc0"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.180444 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:20.180888 21468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.181013 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.181063 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "77e4119e613041a1818753caed54fbc0"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.181097 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:20.195286 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:20.195421 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:20.195528 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:20.195660 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:20.195683 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.195706 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:20.195724 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.198374 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:42469
I20250113 03:06:20.198689 21531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:42469 every 8 connection(s)
I20250113 03:06:20.198844 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:20.200081 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.200914 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.201654 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:20.202200 21532 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40137
I20250113 03:06:20.202255 21532 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:20.202363 21532 heartbeater.cc:510] Master 127.20.18.126:40137 requested a full tablet report, sending...
I20250113 03:06:20.202404 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:20.202514 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:20.202549 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:20.202570 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737580202570 us; error 0 us; skew 500 ppm
I20250113 03:06:20.202601 21403 ts_manager.cc:194] Registered new tserver with Master: 77e4119e613041a1818753caed54fbc0 (127.20.18.65:42469)
I20250113 03:06:20.203190 20553 webserver.cc:458] Webserver started at http://127.20.18.66:37047/ using document root <none> and password file <none>
I20250113 03:06:20.203263 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:20.203310 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:20.203356 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:20.203375 21403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42934
I20250113 03:06:20.203651 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "b2ec9b711c604280832707c21f31c669"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.204447 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:20.204901 21544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.205031 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.205071 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "b2ec9b711c604280832707c21f31c669"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.205116 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:20.221487 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:20.221678 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:20.221805 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:20.221985 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:20.222010 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.222043 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:20.222075 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.224618 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:44359
I20250113 03:06:20.224668 21607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:44359 every 8 connection(s)
I20250113 03:06:20.224961 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:20.226063 21611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.226917 21612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:20.227566 21614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:20.227654 21608 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40137
I20250113 03:06:20.227713 21608 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:20.227819 21608 heartbeater.cc:510] Master 127.20.18.126:40137 requested a full tablet report, sending...
I20250113 03:06:20.228041 21403 ts_manager.cc:194] Registered new tserver with Master: b2ec9b711c604280832707c21f31c669 (127.20.18.66:44359)
I20250113 03:06:20.228358 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:20.228401 21403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42944
I20250113 03:06:20.228480 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:20.228516 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:20.228557 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737580228557 us; error 0 us; skew 500 ppm
I20250113 03:06:20.229161 20553 webserver.cc:458] Webserver started at http://127.20.18.67:37823/ using document root <none> and password file <none>
I20250113 03:06:20.229226 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:20.229269 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:20.229318 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:20.229558 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "419d96c472434b888dfe06f78be1c84e"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.230365 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:20.230749 21619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.230865 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.230906 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "419d96c472434b888dfe06f78be1c84e"
format_stamp: "Formatted at 2025-01-13 03:06:20 on dist-test-slave-d4dl"
I20250113 03:06:20.230948 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:20.238291 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:20.238443 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:20.238561 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:20.238710 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:20.238734 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.238760 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:20.238785 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.241902 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:38623
I20250113 03:06:20.241935 21682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:38623 every 8 connection(s)
I20250113 03:06:20.244544 21683 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40137
I20250113 03:06:20.244601 21683 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:20.244699 21683 heartbeater.cc:510] Master 127.20.18.126:40137 requested a full tablet report, sending...
I20250113 03:06:20.244911 21403 ts_manager.cc:194] Registered new tserver with Master: 419d96c472434b888dfe06f78be1c84e (127.20.18.67:38623)
I20250113 03:06:20.245215 21403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42960
I20250113 03:06:20.245362 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003267665s
I20250113 03:06:20.245685 20553 test_util.cc:274] Using random seed: -1165368651
I20250113 03:06:20.249593 21403 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42976:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:06:20.249693 21403 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:06:20.253598 21496 tablet_service.cc:1467] Processing CreateTablet for tablet 53070ef4ffbc4a728cf796343a22c18b (DEFAULT_TABLE table=test-workload [id=e67394c8f7844be68f68b32d42a8d18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:20.253723 21496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53070ef4ffbc4a728cf796343a22c18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:20.254112 21572 tablet_service.cc:1467] Processing CreateTablet for tablet 53070ef4ffbc4a728cf796343a22c18b (DEFAULT_TABLE table=test-workload [id=e67394c8f7844be68f68b32d42a8d18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:20.254189 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 53070ef4ffbc4a728cf796343a22c18b (DEFAULT_TABLE table=test-workload [id=e67394c8f7844be68f68b32d42a8d18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:20.254226 21572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53070ef4ffbc4a728cf796343a22c18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:20.254267 21647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53070ef4ffbc4a728cf796343a22c18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:20.255178 21698 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: Bootstrap starting.
I20250113 03:06:20.255504 21699 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: Bootstrap starting.
I20250113 03:06:20.255568 21698 tablet_bootstrap.cc:654] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:20.255916 21699 tablet_bootstrap.cc:654] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:20.256234 21698 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: No bootstrap required, opened a new log
I20250113 03:06:20.256259 21700 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: Bootstrap starting.
I20250113 03:06:20.256286 21698 ts_tablet_manager.cc:1397] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:20.256424 21699 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: No bootstrap required, opened a new log
I20250113 03:06:20.256486 21699 ts_tablet_manager.cc:1397] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:20.256423 21698 raft_consensus.cc:357] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.256635 21698 raft_consensus.cc:383] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:20.256685 21700 tablet_bootstrap.cc:654] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:20.256700 21699 raft_consensus.cc:357] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.256688 21698 raft_consensus.cc:738] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e4119e613041a1818753caed54fbc0, State: Initialized, Role: FOLLOWER
I20250113 03:06:20.256788 21699 raft_consensus.cc:383] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:20.256836 21699 raft_consensus.cc:738] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419d96c472434b888dfe06f78be1c84e, State: Initialized, Role: FOLLOWER
I20250113 03:06:20.256840 21698 consensus_queue.cc:260] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.256907 21699 consensus_queue.cc:260] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.257037 21698 ts_tablet_manager.cc:1428] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:20.257107 21532 heartbeater.cc:502] Master 127.20.18.126:40137 was elected leader, sending a full tablet report...
I20250113 03:06:20.257078 21699 ts_tablet_manager.cc:1428] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:20.257261 21683 heartbeater.cc:502] Master 127.20.18.126:40137 was elected leader, sending a full tablet report...
I20250113 03:06:20.257310 21700 tablet_bootstrap.cc:492] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: No bootstrap required, opened a new log
I20250113 03:06:20.257345 21700 ts_tablet_manager.cc:1397] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:20.257511 21700 raft_consensus.cc:357] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.257628 21700 raft_consensus.cc:383] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:20.257673 21700 raft_consensus.cc:738] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ec9b711c604280832707c21f31c669, State: Initialized, Role: FOLLOWER
I20250113 03:06:20.257738 21700 consensus_queue.cc:260] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.257932 21700 ts_tablet_manager.cc:1428] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:06:20.257959 21608 heartbeater.cc:502] Master 127.20.18.126:40137 was elected leader, sending a full tablet report...
I20250113 03:06:20.300971 21706 raft_consensus.cc:491] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:06:20.301051 21706 raft_consensus.cc:513] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.301190 21706 leader_election.cc:290] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e4119e613041a1818753caed54fbc0 (127.20.18.65:42469), 419d96c472434b888dfe06f78be1c84e (127.20.18.67:38623)
I20250113 03:06:20.303953 21506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "b2ec9b711c604280832707c21f31c669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e4119e613041a1818753caed54fbc0" is_pre_election: true
I20250113 03:06:20.303953 21657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "b2ec9b711c604280832707c21f31c669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419d96c472434b888dfe06f78be1c84e" is_pre_election: true
I20250113 03:06:20.304065 21506 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ec9b711c604280832707c21f31c669 in term 0.
I20250113 03:06:20.304067 21657 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ec9b711c604280832707c21f31c669 in term 0.
I20250113 03:06:20.304214 21545 leader_election.cc:304] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e4119e613041a1818753caed54fbc0, b2ec9b711c604280832707c21f31c669; no voters:
I20250113 03:06:20.304293 21706 raft_consensus.cc:2798] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:06:20.304337 21706 raft_consensus.cc:491] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:06:20.304360 21706 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:20.304972 21706 raft_consensus.cc:513] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.305084 21706 leader_election.cc:290] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [CANDIDATE]: Term 1 election: Requested vote from peers 77e4119e613041a1818753caed54fbc0 (127.20.18.65:42469), 419d96c472434b888dfe06f78be1c84e (127.20.18.67:38623)
I20250113 03:06:20.305207 21506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "b2ec9b711c604280832707c21f31c669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e4119e613041a1818753caed54fbc0"
I20250113 03:06:20.305207 21657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "b2ec9b711c604280832707c21f31c669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419d96c472434b888dfe06f78be1c84e"
I20250113 03:06:20.305269 21506 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:20.305286 21657 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:20.305862 21657 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2ec9b711c604280832707c21f31c669 in term 1.
I20250113 03:06:20.305860 21506 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2ec9b711c604280832707c21f31c669 in term 1.
I20250113 03:06:20.306021 21545 leader_election.cc:304] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e4119e613041a1818753caed54fbc0, b2ec9b711c604280832707c21f31c669; no voters:
I20250113 03:06:20.306097 21706 raft_consensus.cc:2798] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:20.306221 21706 raft_consensus.cc:695] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 LEADER]: Becoming Leader. State: Replica: b2ec9b711c604280832707c21f31c669, State: Running, Role: LEADER
I20250113 03:06:20.306284 21706 consensus_queue.cc:237] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:20.306761 21402 catalog_manager.cc:5526] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2ec9b711c604280832707c21f31c669 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "b2ec9b711c604280832707c21f31c669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:20.314422 21657 raft_consensus.cc:1270] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 1 FOLLOWER]: Refusing update from remote peer b2ec9b711c604280832707c21f31c669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:20.314445 21506 raft_consensus.cc:1270] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 1 FOLLOWER]: Refusing update from remote peer b2ec9b711c604280832707c21f31c669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:20.314638 21708 consensus_queue.cc:1035] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:20.314746 21706 consensus_queue.cc:1035] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:30.878481 21733 raft_consensus.cc:988] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: : Instructing follower 77e4119e613041a1818753caed54fbc0 to start an election
I20250113 03:06:30.878556 21733 raft_consensus.cc:1076] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 LEADER]: Signalling peer 77e4119e613041a1818753caed54fbc0 to start an election
I20250113 03:06:30.878785 21505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b"
dest_uuid: "77e4119e613041a1818753caed54fbc0"
from {username='slave'} at 127.0.0.1:40298
I20250113 03:06:30.878851 21505 raft_consensus.cc:491] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:06:30.878870 21505 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:30.879491 21505 raft_consensus.cc:513] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:30.879627 21505 leader_election.cc:290] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [CANDIDATE]: Term 2 election: Requested vote from peers b2ec9b711c604280832707c21f31c669 (127.20.18.66:44359), 419d96c472434b888dfe06f78be1c84e (127.20.18.67:38623)
I20250113 03:06:30.882100 21582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "77e4119e613041a1818753caed54fbc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 15250 } } ignore_live_leader: true dest_uuid: "b2ec9b711c604280832707c21f31c669"
I20250113 03:06:30.882190 21656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53070ef4ffbc4a728cf796343a22c18b" candidate_uuid: "77e4119e613041a1818753caed54fbc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 15250 } } ignore_live_leader: true dest_uuid: "419d96c472434b888dfe06f78be1c84e"
I20250113 03:06:30.882191 21582 raft_consensus.cc:3049] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:06:30.882266 21656 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:30.882284 21582 raft_consensus.cc:738] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2ec9b711c604280832707c21f31c669, State: Running, Role: LEADER
I20250113 03:06:30.882349 21582 consensus_queue.cc:260] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15250, Committed index: 15250, Last appended: 1.15250, Last appended by leader: 15250, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:30.882426 21582 raft_consensus.cc:3054] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:30.882824 21656 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e4119e613041a1818753caed54fbc0 in term 2.
I20250113 03:06:30.883013 21472 leader_election.cc:304] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419d96c472434b888dfe06f78be1c84e, 77e4119e613041a1818753caed54fbc0; no voters:
I20250113 03:06:30.883039 21582 raft_consensus.cc:2463] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e4119e613041a1818753caed54fbc0 in term 2.
I20250113 03:06:30.883210 21764 raft_consensus.cc:2798] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:06:30.883324 21764 raft_consensus.cc:695] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 2 LEADER]: Becoming Leader. State: Replica: 77e4119e613041a1818753caed54fbc0, State: Running, Role: LEADER
I20250113 03:06:30.883375 21764 consensus_queue.cc:237] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15250, Committed index: 15250, Last appended: 1.15250, Last appended by leader: 15250, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } }
I20250113 03:06:30.883772 21401 catalog_manager.cc:5526] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 reported cstate change: term changed from 1 to 2, leader changed from b2ec9b711c604280832707c21f31c669 (127.20.18.66) to 77e4119e613041a1818753caed54fbc0 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "77e4119e613041a1818753caed54fbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e4119e613041a1818753caed54fbc0" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 42469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:30.884359 21656 raft_consensus.cc:1270] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 2 FOLLOWER]: Refusing update from remote peer 77e4119e613041a1818753caed54fbc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 15250. Preceding OpId from leader: term: 2 index: 15252. (index mismatch)
I20250113 03:06:30.884366 21582 raft_consensus.cc:1270] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 2 FOLLOWER]: Refusing update from remote peer 77e4119e613041a1818753caed54fbc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 15250. Preceding OpId from leader: term: 2 index: 15252. (index mismatch)
I20250113 03:06:30.884559 21765 consensus_queue.cc:1035] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419d96c472434b888dfe06f78be1c84e" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15251, Last known committed idx: 15250, Time since last communication: 0.000s
I20250113 03:06:30.884655 21764 consensus_queue.cc:1035] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ec9b711c604280832707c21f31c669" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15251, Last known committed idx: 15250, Time since last communication: 0.000s
I20250113 03:06:30.885510 21768 mvcc.cc:204] Tried to move back new op lower bound from 7113677172261289984 to 7113677172258750464. Current Snapshot: MvccSnapshot[applied={T|T < 7113677172261289984}]
I20250113 03:06:31.916065 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:06:31.918733 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:31.918859 20553 tablet_replica.cc:331] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0: stopping tablet replica
I20250113 03:06:31.918911 20553 raft_consensus.cc:2238] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:06:31.918979 20553 raft_consensus.cc:2267] T 53070ef4ffbc4a728cf796343a22c18b P 77e4119e613041a1818753caed54fbc0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:31.932642 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:06:31.933419 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:06:31.936239 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:31.936358 20553 tablet_replica.cc:331] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669: stopping tablet replica
I20250113 03:06:31.936420 20553 raft_consensus.cc:2238] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:31.936468 20553 raft_consensus.cc:2267] T 53070ef4ffbc4a728cf796343a22c18b P b2ec9b711c604280832707c21f31c669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:31.958148 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:06:31.958930 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:06:31.961899 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:31.962039 20553 tablet_replica.cc:331] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e: stopping tablet replica
I20250113 03:06:31.962086 20553 raft_consensus.cc:2238] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:31.962139 20553 raft_consensus.cc:2267] T 53070ef4ffbc4a728cf796343a22c18b P 419d96c472434b888dfe06f78be1c84e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:31.975754 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:06:31.976577 20553 master.cc:537] Master@127.20.18.126:40137 shutting down...
I20250113 03:06:31.978781 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:31.978844 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:31.978870 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P c9f6e89255294ffda8dfacd242493798: stopping tablet replica
I20250113 03:06:31.990348 20553 master.cc:559] Master@127.20.18.126:40137 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnAbortCommit (11898 ms)
[ RUN ] TxnParticipantITest.TestProxyBasicCalls
I20250113 03:06:32.008551 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:41613
I20250113 03:06:32.008714 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:32.010910 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.011122 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.011888 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.012656 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:32.012754 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:32.012791 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:32.012817 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737592012817 us; error 0 us; skew 500 ppm
I20250113 03:06:32.013406 20553 webserver.cc:458] Webserver started at http://127.20.18.126:39595/ using document root <none> and password file <none>
I20250113 03:06:32.013484 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:32.013525 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:32.013574 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:32.013844 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "eca2c8b777e14db8bad487ee6277dcdc"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.014706 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:32.015093 21791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.015218 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.015259 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "eca2c8b777e14db8bad487ee6277dcdc"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.015304 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:32.024037 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:32.024204 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:32.026404 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:41613
I20250113 03:06:32.034530 21843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:41613 every 8 connection(s)
I20250113 03:06:32.034596 21844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:32.035676 21844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc: Bootstrap starting.
I20250113 03:06:32.035993 21844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:32.036460 21844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc: No bootstrap required, opened a new log
I20250113 03:06:32.036598 21844 raft_consensus.cc:357] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER }
I20250113 03:06:32.036644 21844 raft_consensus.cc:383] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:32.036665 21844 raft_consensus.cc:738] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca2c8b777e14db8bad487ee6277dcdc, State: Initialized, Role: FOLLOWER
I20250113 03:06:32.036717 21844 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER }
I20250113 03:06:32.036746 21844 raft_consensus.cc:397] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:06:32.036764 21844 raft_consensus.cc:491] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:06:32.036789 21844 raft_consensus.cc:3054] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:32.037225 21844 raft_consensus.cc:513] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER }
I20250113 03:06:32.037274 21844 leader_election.cc:304] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca2c8b777e14db8bad487ee6277dcdc; no voters:
I20250113 03:06:32.037362 21844 leader_election.cc:290] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:06:32.037420 21847 raft_consensus.cc:2798] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:32.037490 21844 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:06:32.037549 21847 raft_consensus.cc:695] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 1 LEADER]: Becoming Leader. State: Replica: eca2c8b777e14db8bad487ee6277dcdc, State: Running, Role: LEADER
I20250113 03:06:32.037600 21847 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER }
I20250113 03:06:32.037801 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca2c8b777e14db8bad487ee6277dcdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER } }
I20250113 03:06:32.037806 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca2c8b777e14db8bad487ee6277dcdc. Latest consensus state: current_term: 1 leader_uuid: "eca2c8b777e14db8bad487ee6277dcdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca2c8b777e14db8bad487ee6277dcdc" member_type: VOTER } }
I20250113 03:06:32.037887 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:32.037896 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:32.038105 21852 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:06:32.038249 21852 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:06:32.038749 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:06:32.038765 21852 catalog_manager.cc:1348] Generated new cluster ID: 4cb370acf3ca4008859d63c13d5129d4
I20250113 03:06:32.038807 21852 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:06:32.064041 21852 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:06:32.064469 21852 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:06:32.068913 21852 catalog_manager.cc:5899] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc: Generated new TSK 0
I20250113 03:06:32.068986 21852 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:06:32.070439 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:32.071755 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.072691 20553 server_base.cc:1034] running on GCE node
W20250113 03:06:32.073454 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.074111 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.074218 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:32.074246 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:32.074257 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737592074258 us; error 0 us; skew 500 ppm
I20250113 03:06:32.074851 20553 webserver.cc:458] Webserver started at http://127.20.18.65:46313/ using document root <none> and password file <none>
I20250113 03:06:32.074923 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:32.074955 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:32.074987 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:32.075217 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "085410df9a194ab3b510a79d780d1592"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.075968 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:32.076376 21873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.076516 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.076574 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "085410df9a194ab3b510a79d780d1592"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.076608 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:32.095680 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:32.095834 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:32.095959 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:32.096127 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:32.096153 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.096185 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:32.096205 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.098810 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:43153
I20250113 03:06:32.099161 21936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:43153 every 8 connection(s)
I20250113 03:06:32.099294 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:32.100457 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.101269 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.102066 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.102450 21937 heartbeater.cc:346] Connected to a master server at 127.20.18.126:41613
I20250113 03:06:32.102500 21937 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:32.102591 21937 heartbeater.cc:510] Master 127.20.18.126:41613 requested a full tablet report, sending...
I20250113 03:06:32.102810 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:32.102846 21808 ts_manager.cc:194] Registered new tserver with Master: 085410df9a194ab3b510a79d780d1592 (127.20.18.65:43153)
I20250113 03:06:32.102941 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:32.102972 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:32.102993 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737592102993 us; error 0 us; skew 500 ppm
I20250113 03:06:32.103485 21808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39308
I20250113 03:06:32.103602 20553 webserver.cc:458] Webserver started at http://127.20.18.66:44897/ using document root <none> and password file <none>
I20250113 03:06:32.103677 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:32.103717 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:32.103747 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:32.103965 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "563ec8045d1f45e583ec6ff0f03fc549"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.104732 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:32.105158 21949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.105281 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.105319 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "563ec8045d1f45e583ec6ff0f03fc549"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.105361 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:32.124603 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:32.124763 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:32.124894 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:32.125059 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:32.125082 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.125113 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:32.125134 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.127578 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:36995
I20250113 03:06:32.127638 22012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:36995 every 8 connection(s)
I20250113 03:06:32.127923 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:32.129846 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:32.130033 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.130667 22013 heartbeater.cc:346] Connected to a master server at 127.20.18.126:41613
I20250113 03:06:32.130731 22013 heartbeater.cc:463] Registering TS with master...
W20250113 03:06:32.130808 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:32.130836 22013 heartbeater.cc:510] Master 127.20.18.126:41613 requested a full tablet report, sending...
I20250113 03:06:32.131067 21808 ts_manager.cc:194] Registered new tserver with Master: 563ec8045d1f45e583ec6ff0f03fc549 (127.20.18.66:36995)
I20250113 03:06:32.131374 21808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39320
I20250113 03:06:32.131508 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:32.131610 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:32.131639 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:32.131675 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737592131675 us; error 0 us; skew 500 ppm
I20250113 03:06:32.132222 20553 webserver.cc:458] Webserver started at http://127.20.18.67:41437/ using document root <none> and password file <none>
I20250113 03:06:32.132282 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:32.132313 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:32.132339 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:32.132580 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "54642db476524f2d8a4714857a9089d9"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.133317 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:32.133678 22024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.133795 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.133832 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "54642db476524f2d8a4714857a9089d9"
format_stamp: "Formatted at 2025-01-13 03:06:32 on dist-test-slave-d4dl"
I20250113 03:06:32.133877 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:32.142535 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:32.142704 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:32.142834 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:32.143002 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:32.143026 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.143057 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:32.143082 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.145905 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:36667
I20250113 03:06:32.145956 22087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:36667 every 8 connection(s)
I20250113 03:06:32.148299 22088 heartbeater.cc:346] Connected to a master server at 127.20.18.126:41613
I20250113 03:06:32.148344 22088 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:32.148432 22088 heartbeater.cc:510] Master 127.20.18.126:41613 requested a full tablet report, sending...
I20250113 03:06:32.148630 21808 ts_manager.cc:194] Registered new tserver with Master: 54642db476524f2d8a4714857a9089d9 (127.20.18.67:36667)
I20250113 03:06:32.148936 21808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39326
I20250113 03:06:32.149344 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003273816s
I20250113 03:06:32.149624 20553 test_util.cc:274] Using random seed: -1153464712
I20250113 03:06:32.153441 21808 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39328:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:06:32.153563 21808 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:06:32.157500 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 113120282ed54a2d9008c82d756d70ce (DEFAULT_TABLE table=test-workload [id=a2e5db865d6647558296824aedaf6806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:32.157613 21901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113120282ed54a2d9008c82d756d70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:32.158145 21977 tablet_service.cc:1467] Processing CreateTablet for tablet 113120282ed54a2d9008c82d756d70ce (DEFAULT_TABLE table=test-workload [id=a2e5db865d6647558296824aedaf6806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:32.158249 21977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113120282ed54a2d9008c82d756d70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:32.159046 22103 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: Bootstrap starting.
I20250113 03:06:32.159569 22103 tablet_bootstrap.cc:654] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:32.159765 22104 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: Bootstrap starting.
I20250113 03:06:32.160005 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 113120282ed54a2d9008c82d756d70ce (DEFAULT_TABLE table=test-workload [id=a2e5db865d6647558296824aedaf6806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:32.160097 22052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113120282ed54a2d9008c82d756d70ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:32.160380 22104 tablet_bootstrap.cc:654] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:32.161326 22103 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: No bootstrap required, opened a new log
I20250113 03:06:32.161437 22103 ts_tablet_manager.cc:1397] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:06:32.161317 22107 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: Bootstrap starting.
I20250113 03:06:32.161630 22103 raft_consensus.cc:357] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.161696 22103 raft_consensus.cc:383] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:32.161721 22103 raft_consensus.cc:738] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085410df9a194ab3b510a79d780d1592, State: Initialized, Role: FOLLOWER
I20250113 03:06:32.161815 22103 consensus_queue.cc:260] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.161981 22103 ts_tablet_manager.cc:1428] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.162007 22107 tablet_bootstrap.cc:654] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:32.162060 21937 heartbeater.cc:502] Master 127.20.18.126:41613 was elected leader, sending a full tablet report...
I20250113 03:06:32.162593 22107 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: No bootstrap required, opened a new log
I20250113 03:06:32.162797 22104 tablet_bootstrap.cc:492] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: No bootstrap required, opened a new log
I20250113 03:06:32.162830 22104 ts_tablet_manager.cc:1397] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250113 03:06:32.162938 22104 raft_consensus.cc:357] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.162976 22104 raft_consensus.cc:383] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:32.162988 22104 raft_consensus.cc:738] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563ec8045d1f45e583ec6ff0f03fc549, State: Initialized, Role: FOLLOWER
I20250113 03:06:32.163030 22104 consensus_queue.cc:260] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.163175 22104 ts_tablet_manager.cc:1428] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.163237 22013 heartbeater.cc:502] Master 127.20.18.126:41613 was elected leader, sending a full tablet report...
I20250113 03:06:32.162663 22107 ts_tablet_manager.cc:1397] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:32.163540 22107 raft_consensus.cc:357] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.163604 22107 raft_consensus.cc:383] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:32.163621 22107 raft_consensus.cc:738] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54642db476524f2d8a4714857a9089d9, State: Initialized, Role: FOLLOWER
I20250113 03:06:32.163684 22107 consensus_queue.cc:260] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.163853 22107 ts_tablet_manager.cc:1428] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:32.164564 22088 heartbeater.cc:502] Master 127.20.18.126:41613 was elected leader, sending a full tablet report...
I20250113 03:06:32.165661 22110 raft_consensus.cc:491] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:06:32.165714 22110 raft_consensus.cc:513] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.165863 22110 leader_election.cc:290] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085410df9a194ab3b510a79d780d1592 (127.20.18.65:43153), 54642db476524f2d8a4714857a9089d9 (127.20.18.67:36667)
I20250113 03:06:32.168206 21911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "563ec8045d1f45e583ec6ff0f03fc549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085410df9a194ab3b510a79d780d1592" is_pre_election: true
I20250113 03:06:32.168272 21911 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563ec8045d1f45e583ec6ff0f03fc549 in term 0.
I20250113 03:06:32.168248 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "563ec8045d1f45e583ec6ff0f03fc549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54642db476524f2d8a4714857a9089d9" is_pre_election: true
I20250113 03:06:32.168319 22062 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 563ec8045d1f45e583ec6ff0f03fc549 in term 0.
I20250113 03:06:32.168399 21951 leader_election.cc:304] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085410df9a194ab3b510a79d780d1592, 563ec8045d1f45e583ec6ff0f03fc549; no voters:
I20250113 03:06:32.168474 22110 raft_consensus.cc:2798] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:06:32.168504 22110 raft_consensus.cc:491] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:06:32.168548 22110 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:32.169023 22110 raft_consensus.cc:513] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.169134 22110 leader_election.cc:290] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [CANDIDATE]: Term 1 election: Requested vote from peers 085410df9a194ab3b510a79d780d1592 (127.20.18.65:43153), 54642db476524f2d8a4714857a9089d9 (127.20.18.67:36667)
I20250113 03:06:32.169263 21911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "563ec8045d1f45e583ec6ff0f03fc549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085410df9a194ab3b510a79d780d1592"
I20250113 03:06:32.169281 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "563ec8045d1f45e583ec6ff0f03fc549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54642db476524f2d8a4714857a9089d9"
I20250113 03:06:32.169319 21911 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:32.169322 22062 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:32.169919 21911 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563ec8045d1f45e583ec6ff0f03fc549 in term 1.
I20250113 03:06:32.169922 22062 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563ec8045d1f45e583ec6ff0f03fc549 in term 1.
I20250113 03:06:32.170079 21951 leader_election.cc:304] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085410df9a194ab3b510a79d780d1592, 563ec8045d1f45e583ec6ff0f03fc549; no voters:
I20250113 03:06:32.170171 22110 raft_consensus.cc:2798] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:32.170287 22110 raft_consensus.cc:695] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 LEADER]: Becoming Leader. State: Replica: 563ec8045d1f45e583ec6ff0f03fc549, State: Running, Role: LEADER
I20250113 03:06:32.170369 22110 consensus_queue.cc:237] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:32.170820 21808 catalog_manager.cc:5526] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 reported cstate change: term changed from 0 to 1, leader changed from <none> to 563ec8045d1f45e583ec6ff0f03fc549 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "563ec8045d1f45e583ec6ff0f03fc549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:32.177824 22062 raft_consensus.cc:1270] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 1 FOLLOWER]: Refusing update from remote peer 563ec8045d1f45e583ec6ff0f03fc549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:32.177847 21911 raft_consensus.cc:1270] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 1 FOLLOWER]: Refusing update from remote peer 563ec8045d1f45e583ec6ff0f03fc549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:32.178053 22113 consensus_queue.cc:1035] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:32.178117 22110 consensus_queue.cc:1035] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:42.612763 22150 raft_consensus.cc:988] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: : Instructing follower 085410df9a194ab3b510a79d780d1592 to start an election
I20250113 03:06:42.612848 22150 raft_consensus.cc:1076] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 LEADER]: Signalling peer 085410df9a194ab3b510a79d780d1592 to start an election
I20250113 03:06:42.612989 21910 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce"
dest_uuid: "085410df9a194ab3b510a79d780d1592"
from {username='slave'} at 127.0.0.1:42842
I20250113 03:06:42.613044 21910 raft_consensus.cc:491] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:06:42.613080 21910 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:42.613857 21910 raft_consensus.cc:513] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:42.614008 21910 leader_election.cc:290] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [CANDIDATE]: Term 2 election: Requested vote from peers 563ec8045d1f45e583ec6ff0f03fc549 (127.20.18.66:36995), 54642db476524f2d8a4714857a9089d9 (127.20.18.67:36667)
I20250113 03:06:42.616566 21987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "085410df9a194ab3b510a79d780d1592" candidate_term: 2 candidate_status { last_received { term: 1 index: 15516 } } ignore_live_leader: true dest_uuid: "563ec8045d1f45e583ec6ff0f03fc549"
I20250113 03:06:42.616647 21987 raft_consensus.cc:3049] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:06:42.616639 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113120282ed54a2d9008c82d756d70ce" candidate_uuid: "085410df9a194ab3b510a79d780d1592" candidate_term: 2 candidate_status { last_received { term: 1 index: 15516 } } ignore_live_leader: true dest_uuid: "54642db476524f2d8a4714857a9089d9"
I20250113 03:06:42.616747 21987 raft_consensus.cc:738] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 563ec8045d1f45e583ec6ff0f03fc549, State: Running, Role: LEADER
I20250113 03:06:42.616770 22062 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:42.616811 21987 consensus_queue.cc:260] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15516, Committed index: 15516, Last appended: 1.15516, Last appended by leader: 15516, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:42.616899 21987 raft_consensus.cc:3054] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:42.617463 22062 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085410df9a194ab3b510a79d780d1592 in term 2.
I20250113 03:06:42.617501 21987 raft_consensus.cc:2463] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085410df9a194ab3b510a79d780d1592 in term 2.
I20250113 03:06:42.617660 21877 leader_election.cc:304] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 085410df9a194ab3b510a79d780d1592, 54642db476524f2d8a4714857a9089d9, 563ec8045d1f45e583ec6ff0f03fc549; no voters:
I20250113 03:06:42.617847 22168 raft_consensus.cc:2798] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:06:42.617957 22168 raft_consensus.cc:695] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 2 LEADER]: Becoming Leader. State: Replica: 085410df9a194ab3b510a79d780d1592, State: Running, Role: LEADER
I20250113 03:06:42.618022 22168 consensus_queue.cc:237] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15516, Committed index: 15516, Last appended: 1.15516, Last appended by leader: 15516, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } }
I20250113 03:06:42.618415 21806 catalog_manager.cc:5526] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 reported cstate change: term changed from 1 to 2, leader changed from 563ec8045d1f45e583ec6ff0f03fc549 (127.20.18.66) to 085410df9a194ab3b510a79d780d1592 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "085410df9a194ab3b510a79d780d1592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085410df9a194ab3b510a79d780d1592" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:42.630266 21987 raft_consensus.cc:1270] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 2 FOLLOWER]: Refusing update from remote peer 085410df9a194ab3b510a79d780d1592: Log matching property violated. Preceding OpId in replica: term: 1 index: 15516. Preceding OpId from leader: term: 2 index: 15518. (index mismatch)
I20250113 03:06:42.630266 22062 raft_consensus.cc:1270] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 2 FOLLOWER]: Refusing update from remote peer 085410df9a194ab3b510a79d780d1592: Log matching property violated. Preceding OpId in replica: term: 1 index: 15516. Preceding OpId from leader: term: 2 index: 15518. (index mismatch)
I20250113 03:06:42.630471 22168 consensus_queue.cc:1035] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563ec8045d1f45e583ec6ff0f03fc549" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15517, Last known committed idx: 15516, Time since last communication: 0.000s
I20250113 03:06:42.630586 22169 consensus_queue.cc:1035] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54642db476524f2d8a4714857a9089d9" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15517, Last known committed idx: 15516, Time since last communication: 0.000s
I20250113 03:06:42.633635 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:06:42.635763 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:42.635856 20553 tablet_replica.cc:331] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592: stopping tablet replica
I20250113 03:06:42.635907 20553 raft_consensus.cc:2238] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:06:42.635958 20553 raft_consensus.cc:2267] T 113120282ed54a2d9008c82d756d70ce P 085410df9a194ab3b510a79d780d1592 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:42.649581 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:06:42.650259 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:06:42.653090 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:42.653205 20553 tablet_replica.cc:331] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549: stopping tablet replica
I20250113 03:06:42.653260 20553 raft_consensus.cc:2238] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:42.653302 20553 raft_consensus.cc:2267] T 113120282ed54a2d9008c82d756d70ce P 563ec8045d1f45e583ec6ff0f03fc549 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:42.673951 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:06:42.674691 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:06:42.677367 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:42.677469 20553 tablet_replica.cc:331] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9: stopping tablet replica
I20250113 03:06:42.677524 20553 raft_consensus.cc:2238] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:42.677567 20553 raft_consensus.cc:2267] T 113120282ed54a2d9008c82d756d70ce P 54642db476524f2d8a4714857a9089d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:42.690650 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:06:42.691287 20553 master.cc:537] Master@127.20.18.126:41613 shutting down...
I20250113 03:06:42.693424 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:42.693481 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:42.693511 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P eca2c8b777e14db8bad487ee6277dcdc: stopping tablet replica
I20250113 03:06:42.705163 20553 master.cc:559] Master@127.20.18.126:41613 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyBasicCalls (10715 ms)
[ RUN ] TxnParticipantITest.TestBeginCommitAfterFinalize
I20250113 03:06:42.723922 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:34767
I20250113 03:06:42.724061 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:42.725432 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:42.726336 20553 server_base.cc:1034] running on GCE node
W20250113 03:06:42.727066 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.727715 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:42.727830 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:42.727867 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:42.727885 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737602727885 us; error 0 us; skew 500 ppm
I20250113 03:06:42.728454 20553 webserver.cc:458] Webserver started at http://127.20.18.126:42773/ using document root <none> and password file <none>
I20250113 03:06:42.728559 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:42.728607 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:42.728654 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:42.728896 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "f376d4199be7433ead54f82e4b9da686"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.729679 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:42.730054 22185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.730168 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.730207 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "f376d4199be7433ead54f82e4b9da686"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.730252 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:42.746582 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:42.746745 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:42.749020 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:34767
I20250113 03:06:42.760087 22237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:34767 every 8 connection(s)
I20250113 03:06:42.760238 22238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:42.761217 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686: Bootstrap starting.
I20250113 03:06:42.761502 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:42.762020 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686: No bootstrap required, opened a new log
I20250113 03:06:42.762125 22238 raft_consensus.cc:357] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER }
I20250113 03:06:42.762169 22238 raft_consensus.cc:383] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:42.762187 22238 raft_consensus.cc:738] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f376d4199be7433ead54f82e4b9da686, State: Initialized, Role: FOLLOWER
I20250113 03:06:42.762260 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER }
I20250113 03:06:42.762291 22238 raft_consensus.cc:397] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:06:42.762312 22238 raft_consensus.cc:491] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:06:42.762336 22238 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:42.762748 22238 raft_consensus.cc:513] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER }
I20250113 03:06:42.762794 22238 leader_election.cc:304] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f376d4199be7433ead54f82e4b9da686; no voters:
I20250113 03:06:42.762900 22238 leader_election.cc:290] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:06:42.762943 22241 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:42.763022 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:06:42.763100 22241 raft_consensus.cc:695] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 1 LEADER]: Becoming Leader. State: Replica: f376d4199be7433ead54f82e4b9da686, State: Running, Role: LEADER
I20250113 03:06:42.763162 22241 consensus_queue.cc:237] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER }
I20250113 03:06:42.763345 22242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f376d4199be7433ead54f82e4b9da686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER } }
I20250113 03:06:42.763414 22242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:42.763604 22243 sys_catalog.cc:455] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f376d4199be7433ead54f82e4b9da686. Latest consensus state: current_term: 1 leader_uuid: "f376d4199be7433ead54f82e4b9da686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f376d4199be7433ead54f82e4b9da686" member_type: VOTER } }
I20250113 03:06:42.763686 22247 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:06:42.763685 22243 sys_catalog.cc:458] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:42.763839 22247 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:06:42.764181 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:06:42.764434 22247 catalog_manager.cc:1348] Generated new cluster ID: 5fa32398c5bc432087a855edd896a3cc
I20250113 03:06:42.764467 22247 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:06:42.771939 22247 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:06:42.772329 22247 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:06:42.775528 22247 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686: Generated new TSK 0
I20250113 03:06:42.775580 22247 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:06:42.779860 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:42.781731 22259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.781947 22260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.782773 22262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:42.783458 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:42.783537 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:42.783564 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:42.783576 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737602783576 us; error 0 us; skew 500 ppm
I20250113 03:06:42.784148 20553 webserver.cc:458] Webserver started at http://127.20.18.65:34317/ using document root <none> and password file <none>
I20250113 03:06:42.784221 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:42.784260 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:42.784303 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:42.784512 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "63b2e493f480456e8293b4c13208899d"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.785305 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:42.785669 22267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.785753 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.785790 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "63b2e493f480456e8293b4c13208899d"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.785832 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:42.799428 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:42.799567 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:42.799678 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:42.799840 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:42.799865 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.799890 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:42.799916 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.802335 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:34309
I20250113 03:06:42.802358 22330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:34309 every 8 connection(s)
I20250113 03:06:42.802630 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:42.803594 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.804378 22336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.805099 22338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:42.805647 22331 heartbeater.cc:346] Connected to a master server at 127.20.18.126:34767
I20250113 03:06:42.805691 22331 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:42.805725 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:42.805820 22331 heartbeater.cc:510] Master 127.20.18.126:34767 requested a full tablet report, sending...
I20250113 03:06:42.805836 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:42.805859 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:42.805879 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737602805879 us; error 0 us; skew 500 ppm
I20250113 03:06:42.806095 22202 ts_manager.cc:194] Registered new tserver with Master: 63b2e493f480456e8293b4c13208899d (127.20.18.65:34309)
I20250113 03:06:42.806445 20553 webserver.cc:458] Webserver started at http://127.20.18.66:34853/ using document root <none> and password file <none>
I20250113 03:06:42.806514 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:42.806550 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:42.806587 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:42.806744 22202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58408
I20250113 03:06:42.806890 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "7c797d3e385a463babe8519b26e0d088"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.807726 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:42.808087 22343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.808192 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.808226 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "7c797d3e385a463babe8519b26e0d088"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.808261 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:42.817085 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:42.817220 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:42.817325 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:42.817476 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:42.817498 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.817523 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:42.817546 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.820019 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:41015
I20250113 03:06:42.820075 22406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:41015 every 8 connection(s)
I20250113 03:06:42.820329 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:42.821460 22410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.822245 22413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:42.822969 22411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:42.822986 22407 heartbeater.cc:346] Connected to a master server at 127.20.18.126:34767
I20250113 03:06:42.823048 22407 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:42.823143 22407 heartbeater.cc:510] Master 127.20.18.126:34767 requested a full tablet report, sending...
I20250113 03:06:42.823346 22202 ts_manager.cc:194] Registered new tserver with Master: 7c797d3e385a463babe8519b26e0d088 (127.20.18.66:41015)
I20250113 03:06:42.823659 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:42.823678 22202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58422
I20250113 03:06:42.823784 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:42.823818 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:42.823837 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737602823837 us; error 0 us; skew 500 ppm
I20250113 03:06:42.824445 20553 webserver.cc:458] Webserver started at http://127.20.18.67:33791/ using document root <none> and password file <none>
I20250113 03:06:42.824504 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:42.824558 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:42.824600 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:42.824815 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "f7aa16c2a6e34ed6acd5f96af26b319d"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.825547 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:42.825914 22418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.826018 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.826057 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "f7aa16c2a6e34ed6acd5f96af26b319d"
format_stamp: "Formatted at 2025-01-13 03:06:42 on dist-test-slave-d4dl"
I20250113 03:06:42.826102 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:42.832960 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:42.833112 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:42.833235 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:42.833384 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:42.833408 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.833434 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:42.833458 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.836416 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:45679
I20250113 03:06:42.836462 22481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:45679 every 8 connection(s)
I20250113 03:06:42.838946 22482 heartbeater.cc:346] Connected to a master server at 127.20.18.126:34767
I20250113 03:06:42.838994 22482 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:42.839077 22482 heartbeater.cc:510] Master 127.20.18.126:34767 requested a full tablet report, sending...
I20250113 03:06:42.839275 22202 ts_manager.cc:194] Registered new tserver with Master: f7aa16c2a6e34ed6acd5f96af26b319d (127.20.18.67:45679)
I20250113 03:06:42.839548 22202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58436
I20250113 03:06:42.839882 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003288471s
I20250113 03:06:42.840178 20553 test_util.cc:274] Using random seed: -1142774158
I20250113 03:06:42.843925 22202 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58448:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:06:42.844033 22202 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:06:42.848220 22446 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 (DEFAULT_TABLE table=test-workload [id=1152639fb5ef4efaa6918285d9dfef03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:42.848320 22446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:42.848603 22295 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 (DEFAULT_TABLE table=test-workload [id=1152639fb5ef4efaa6918285d9dfef03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:42.848683 22295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:42.848912 22371 tablet_service.cc:1467] Processing CreateTablet for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 (DEFAULT_TABLE table=test-workload [id=1152639fb5ef4efaa6918285d9dfef03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:42.848990 22371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:42.849625 22497 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: Bootstrap starting.
I20250113 03:06:42.849890 22498 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: Bootstrap starting.
I20250113 03:06:42.850073 22497 tablet_bootstrap.cc:654] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:42.850114 22499 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: Bootstrap starting.
I20250113 03:06:42.850363 22498 tablet_bootstrap.cc:654] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:42.850513 22499 tablet_bootstrap.cc:654] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:42.850919 22498 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: No bootstrap required, opened a new log
I20250113 03:06:42.850919 22497 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: No bootstrap required, opened a new log
I20250113 03:06:42.850982 22497 ts_tablet_manager.cc:1397] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:42.850981 22498 ts_tablet_manager.cc:1397] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:42.851020 22499 tablet_bootstrap.cc:492] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: No bootstrap required, opened a new log
I20250113 03:06:42.851094 22499 ts_tablet_manager.cc:1397] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:42.851161 22497 raft_consensus.cc:357] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851210 22497 raft_consensus.cc:383] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:42.851223 22499 raft_consensus.cc:357] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851269 22497 raft_consensus.cc:738] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7aa16c2a6e34ed6acd5f96af26b319d, State: Initialized, Role: FOLLOWER
I20250113 03:06:42.851209 22498 raft_consensus.cc:357] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851280 22499 raft_consensus.cc:383] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:42.851352 22499 raft_consensus.cc:738] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c797d3e385a463babe8519b26e0d088, State: Initialized, Role: FOLLOWER
I20250113 03:06:42.851336 22498 raft_consensus.cc:383] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:42.851344 22497 consensus_queue.cc:260] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851435 22498 raft_consensus.cc:738] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b2e493f480456e8293b4c13208899d, State: Initialized, Role: FOLLOWER
I20250113 03:06:42.851455 22499 consensus_queue.cc:260] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851475 22498 consensus_queue.cc:260] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.851547 22497 ts_tablet_manager.cc:1428] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:06:42.851590 22499 ts_tablet_manager.cc:1428] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:42.851624 22482 heartbeater.cc:502] Master 127.20.18.126:34767 was elected leader, sending a full tablet report...
I20250113 03:06:42.851591 22498 ts_tablet_manager.cc:1428] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:06:42.851689 22331 heartbeater.cc:502] Master 127.20.18.126:34767 was elected leader, sending a full tablet report...
I20250113 03:06:42.851708 22407 heartbeater.cc:502] Master 127.20.18.126:34767 was elected leader, sending a full tablet report...
I20250113 03:06:42.863253 22503 raft_consensus.cc:491] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:06:42.863309 22503 raft_consensus.cc:513] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.863425 22503 leader_election.cc:290] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c797d3e385a463babe8519b26e0d088 (127.20.18.66:41015), 63b2e493f480456e8293b4c13208899d (127.20.18.65:34309)
I20250113 03:06:42.865653 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2e493f480456e8293b4c13208899d" is_pre_election: true
I20250113 03:06:42.865731 22305 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7aa16c2a6e34ed6acd5f96af26b319d in term 0.
I20250113 03:06:42.865804 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c797d3e385a463babe8519b26e0d088" is_pre_election: true
I20250113 03:06:42.865860 22381 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7aa16c2a6e34ed6acd5f96af26b319d in term 0.
I20250113 03:06:42.865952 22420 leader_election.cc:304] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2e493f480456e8293b4c13208899d, f7aa16c2a6e34ed6acd5f96af26b319d; no voters:
I20250113 03:06:42.866102 22503 raft_consensus.cc:2798] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:06:42.866134 22503 raft_consensus.cc:491] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:06:42.866146 22503 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:42.866626 22503 raft_consensus.cc:513] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.866725 22503 leader_election.cc:290] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [CANDIDATE]: Term 1 election: Requested vote from peers 7c797d3e385a463babe8519b26e0d088 (127.20.18.66:41015), 63b2e493f480456e8293b4c13208899d (127.20.18.65:34309)
I20250113 03:06:42.866837 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c797d3e385a463babe8519b26e0d088"
I20250113 03:06:42.866837 22305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b2e493f480456e8293b4c13208899d"
I20250113 03:06:42.866928 22381 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:42.866935 22305 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:42.867561 22381 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7aa16c2a6e34ed6acd5f96af26b319d in term 1.
I20250113 03:06:42.867561 22305 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7aa16c2a6e34ed6acd5f96af26b319d in term 1.
I20250113 03:06:42.867733 22420 leader_election.cc:304] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2e493f480456e8293b4c13208899d, f7aa16c2a6e34ed6acd5f96af26b319d; no voters:
I20250113 03:06:42.867813 22503 raft_consensus.cc:2798] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:42.867928 22503 raft_consensus.cc:695] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 LEADER]: Becoming Leader. State: Replica: f7aa16c2a6e34ed6acd5f96af26b319d, State: Running, Role: LEADER
I20250113 03:06:42.867972 22503 consensus_queue.cc:237] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:42.868394 22201 catalog_manager.cc:5526] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7aa16c2a6e34ed6acd5f96af26b319d (127.20.18.67). New cstate: current_term: 1 leader_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } health_report { overall_health: HEALTHY } } }
I20250113 03:06:42.882396 22305 raft_consensus.cc:1270] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 1 FOLLOWER]: Refusing update from remote peer f7aa16c2a6e34ed6acd5f96af26b319d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:42.882512 22381 raft_consensus.cc:1270] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 1 FOLLOWER]: Refusing update from remote peer f7aa16c2a6e34ed6acd5f96af26b319d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:42.882669 22507 consensus_queue.cc:1035] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:42.882763 22503 consensus_queue.cc:1035] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:53.505532 22557 raft_consensus.cc:988] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: : Instructing follower 63b2e493f480456e8293b4c13208899d to start an election
I20250113 03:06:53.505646 22550 raft_consensus.cc:1076] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 LEADER]: Signalling peer 63b2e493f480456e8293b4c13208899d to start an election
I20250113 03:06:53.505867 22305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0"
dest_uuid: "63b2e493f480456e8293b4c13208899d"
from {username='slave'} at 127.0.0.1:53374
I20250113 03:06:53.506071 22305 raft_consensus.cc:491] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:06:53.506119 22305 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:53.506705 22305 raft_consensus.cc:513] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:53.506856 22305 leader_election.cc:290] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [CANDIDATE]: Term 2 election: Requested vote from peers 7c797d3e385a463babe8519b26e0d088 (127.20.18.66:41015), f7aa16c2a6e34ed6acd5f96af26b319d (127.20.18.67:45679)
I20250113 03:06:53.509968 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "63b2e493f480456e8293b4c13208899d" candidate_term: 2 candidate_status { last_received { term: 1 index: 15305 } } ignore_live_leader: true dest_uuid: "7c797d3e385a463babe8519b26e0d088"
I20250113 03:06:53.510015 22456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c2c6e0ab4d4e158a8d9a1e9bf7a6a0" candidate_uuid: "63b2e493f480456e8293b4c13208899d" candidate_term: 2 candidate_status { last_received { term: 1 index: 15305 } } ignore_live_leader: true dest_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d"
I20250113 03:06:53.510051 22381 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:53.510057 22456 raft_consensus.cc:3049] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:06:53.510109 22456 raft_consensus.cc:738] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7aa16c2a6e34ed6acd5f96af26b319d, State: Running, Role: LEADER
I20250113 03:06:53.510166 22456 consensus_queue.cc:260] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15305, Committed index: 15305, Last appended: 1.15305, Last appended by leader: 15305, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:53.510248 22456 raft_consensus.cc:3054] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:06:53.510805 22381 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b2e493f480456e8293b4c13208899d in term 2.
I20250113 03:06:53.510913 22456 raft_consensus.cc:2463] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b2e493f480456e8293b4c13208899d in term 2.
I20250113 03:06:53.511073 22269 leader_election.cc:304] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b2e493f480456e8293b4c13208899d, f7aa16c2a6e34ed6acd5f96af26b319d; no voters:
I20250113 03:06:53.511266 22562 raft_consensus.cc:2798] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:06:53.511392 22562 raft_consensus.cc:695] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 2 LEADER]: Becoming Leader. State: Replica: 63b2e493f480456e8293b4c13208899d, State: Running, Role: LEADER
I20250113 03:06:53.511448 22562 consensus_queue.cc:237] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15305, Committed index: 15305, Last appended: 1.15305, Last appended by leader: 15305, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } }
I20250113 03:06:53.511838 22202 catalog_manager.cc:5526] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d reported cstate change: term changed from 1 to 2, leader changed from f7aa16c2a6e34ed6acd5f96af26b319d (127.20.18.67) to 63b2e493f480456e8293b4c13208899d (127.20.18.65). New cstate: current_term: 2 leader_uuid: "63b2e493f480456e8293b4c13208899d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b2e493f480456e8293b4c13208899d" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 34309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:53.523519 22456 raft_consensus.cc:1270] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 2 FOLLOWER]: Refusing update from remote peer 63b2e493f480456e8293b4c13208899d: Log matching property violated. Preceding OpId in replica: term: 1 index: 15305. Preceding OpId from leader: term: 2 index: 15307. (index mismatch)
I20250113 03:06:53.523522 22381 raft_consensus.cc:1270] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 2 FOLLOWER]: Refusing update from remote peer 63b2e493f480456e8293b4c13208899d: Log matching property violated. Preceding OpId in replica: term: 1 index: 15305. Preceding OpId from leader: term: 2 index: 15307. (index mismatch)
I20250113 03:06:53.523775 22563 consensus_queue.cc:1035] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c797d3e385a463babe8519b26e0d088" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15306, Last known committed idx: 15305, Time since last communication: 0.000s
I20250113 03:06:53.523876 22562 consensus_queue.cc:1035] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7aa16c2a6e34ed6acd5f96af26b319d" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15306, Last known committed idx: 15305, Time since last communication: 0.000s
W20250113 03:06:53.526903 22570 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32986: Illegal state: Transaction 0 commit already in progress
I20250113 03:06:53.527176 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:06:53.529654 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:53.529750 20553 tablet_replica.cc:331] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d: stopping tablet replica
I20250113 03:06:53.529805 20553 raft_consensus.cc:2238] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:06:53.529868 20553 raft_consensus.cc:2267] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 63b2e493f480456e8293b4c13208899d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:53.544294 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:06:53.545188 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:06:53.548321 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:53.548422 20553 tablet_replica.cc:331] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088: stopping tablet replica
I20250113 03:06:53.548480 20553 raft_consensus.cc:2238] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:53.548542 20553 raft_consensus.cc:2267] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P 7c797d3e385a463babe8519b26e0d088 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:53.567759 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:06:53.568657 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:06:53.571751 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:06:53.571857 20553 tablet_replica.cc:331] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d: stopping tablet replica
I20250113 03:06:53.571919 20553 raft_consensus.cc:2238] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:06:53.571954 20553 raft_consensus.cc:2267] T 14c2c6e0ab4d4e158a8d9a1e9bf7a6a0 P f7aa16c2a6e34ed6acd5f96af26b319d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:53.593730 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:06:53.594527 20553 master.cc:537] Master@127.20.18.126:34767 shutting down...
I20250113 03:06:53.597095 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:06:53.597167 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:06:53.597191 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P f376d4199be7433ead54f82e4b9da686: stopping tablet replica
I20250113 03:06:53.608839 20553 master.cc:559] Master@127.20.18.126:34767 shutdown complete.
[ OK ] TxnParticipantITest.TestBeginCommitAfterFinalize (10903 ms)
[ RUN ] TxnParticipantITest.TestProxyErrorWhenNotBegun
I20250113 03:06:53.627422 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:37419
I20250113 03:06:53.627578 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:53.629277 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.630177 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.631544 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:53.631585 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:53.631662 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:53.631701 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:53.631723 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737613631723 us; error 0 us; skew 500 ppm
I20250113 03:06:53.632287 20553 webserver.cc:458] Webserver started at http://127.20.18.126:36991/ using document root <none> and password file <none>
I20250113 03:06:53.632345 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:53.632375 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:53.632406 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:53.632719 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "c4a70472319b49338d51e96fd4bb136a"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.633567 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:53.633942 22586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.634058 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.634099 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "c4a70472319b49338d51e96fd4bb136a"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.634142 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:53.666960 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:53.667152 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:53.669492 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:37419
I20250113 03:06:53.671422 22639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:53.671450 22638 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:37419 every 8 connection(s)
I20250113 03:06:53.672334 22639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a: Bootstrap starting.
I20250113 03:06:53.672631 22639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:53.673048 22639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a: No bootstrap required, opened a new log
I20250113 03:06:53.673158 22639 raft_consensus.cc:357] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER }
I20250113 03:06:53.673200 22639 raft_consensus.cc:383] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:53.673223 22639 raft_consensus.cc:738] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a70472319b49338d51e96fd4bb136a, State: Initialized, Role: FOLLOWER
I20250113 03:06:53.673267 22639 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER }
I20250113 03:06:53.673308 22639 raft_consensus.cc:397] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:06:53.673333 22639 raft_consensus.cc:491] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:06:53.673352 22639 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:53.673763 22639 raft_consensus.cc:513] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER }
I20250113 03:06:53.673811 22639 leader_election.cc:304] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4a70472319b49338d51e96fd4bb136a; no voters:
I20250113 03:06:53.673893 22639 leader_election.cc:290] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:06:53.673993 22642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:53.674060 22639 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:06:53.674147 22642 raft_consensus.cc:695] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 1 LEADER]: Becoming Leader. State: Replica: c4a70472319b49338d51e96fd4bb136a, State: Running, Role: LEADER
I20250113 03:06:53.674223 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER }
I20250113 03:06:53.674435 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4a70472319b49338d51e96fd4bb136a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER } }
I20250113 03:06:53.674436 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4a70472319b49338d51e96fd4bb136a. Latest consensus state: current_term: 1 leader_uuid: "c4a70472319b49338d51e96fd4bb136a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a70472319b49338d51e96fd4bb136a" member_type: VOTER } }
I20250113 03:06:53.674517 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:53.674525 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [sys.catalog]: This master's current role is: LEADER
I20250113 03:06:53.674791 22648 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:06:53.674911 22648 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:06:53.675336 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:06:53.675521 22648 catalog_manager.cc:1348] Generated new cluster ID: f3aebd30d09b4cd19fe1441f1f5ca799
I20250113 03:06:53.675560 22648 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:06:53.690569 22648 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:06:53.690989 22648 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:06:53.694710 22648 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a: Generated new TSK 0
I20250113 03:06:53.694790 22648 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:06:53.707187 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:53.708609 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.709518 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.710260 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:53.710997 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:53.711091 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:53.711123 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:53.711143 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737613711142 us; error 0 us; skew 500 ppm
I20250113 03:06:53.711784 20553 webserver.cc:458] Webserver started at http://127.20.18.65:38767/ using document root <none> and password file <none>
I20250113 03:06:53.711869 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:53.711915 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:53.711966 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:53.712242 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "233241ab1fe34f85a6218cf643dc887a"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.713100 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:53.713495 22668 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.713608 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.713649 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "233241ab1fe34f85a6218cf643dc887a"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.713693 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:53.732944 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:53.733144 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:53.733338 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:53.733512 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:53.733536 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.733557 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:53.733573 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.736186 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:45709
I20250113 03:06:53.736317 22731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:45709 every 8 connection(s)
I20250113 03:06:53.736603 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:06:53.737953 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.738909 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.739673 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:53.739807 22732 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37419
I20250113 03:06:53.739866 22732 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:53.739984 22732 heartbeater.cc:510] Master 127.20.18.126:37419 requested a full tablet report, sending...
I20250113 03:06:53.740242 22603 ts_manager.cc:194] Registered new tserver with Master: 233241ab1fe34f85a6218cf643dc887a (127.20.18.65:45709)
I20250113 03:06:53.740371 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:53.740464 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:53.740497 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:53.740514 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737613740514 us; error 0 us; skew 500 ppm
I20250113 03:06:53.740891 22603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38368
I20250113 03:06:53.741319 20553 webserver.cc:458] Webserver started at http://127.20.18.66:43279/ using document root <none> and password file <none>
I20250113 03:06:53.741389 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:53.741425 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:53.741452 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:53.741678 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "4bbe18130c5446d29cbaa535c4017c0e"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.742455 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:53.742861 22744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.742960 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.743006 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "4bbe18130c5446d29cbaa535c4017c0e"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.743048 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:53.777067 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:53.777256 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:53.777403 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:53.777613 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:53.777642 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.777676 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:53.777705 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.780325 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:34831
I20250113 03:06:53.780848 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:06:53.781257 22807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:34831 every 8 connection(s)
W20250113 03:06:53.782444 22810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.783274 22813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:06:53.784072 22811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:06:53.784313 22808 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37419
I20250113 03:06:53.784359 22808 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:53.784437 22808 heartbeater.cc:510] Master 127.20.18.126:37419 requested a full tablet report, sending...
I20250113 03:06:53.784726 22603 ts_manager.cc:194] Registered new tserver with Master: 4bbe18130c5446d29cbaa535c4017c0e (127.20.18.66:34831)
I20250113 03:06:53.784772 20553 server_base.cc:1034] running on GCE node
I20250113 03:06:53.784859 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:06:53.784891 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:06:53.784904 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737613784905 us; error 0 us; skew 500 ppm
I20250113 03:06:53.785112 22603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38370
I20250113 03:06:53.785542 20553 webserver.cc:458] Webserver started at http://127.20.18.67:46827/ using document root <none> and password file <none>
I20250113 03:06:53.785612 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:06:53.785657 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:06:53.785707 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:06:53.786010 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "b1054e18ce454328aae3032e2504eaa6"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.786788 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:53.787217 22819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.787329 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.787369 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "b1054e18ce454328aae3032e2504eaa6"
format_stamp: "Formatted at 2025-01-13 03:06:53 on dist-test-slave-d4dl"
I20250113 03:06:53.787411 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:06:53.792658 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:06:53.792809 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:06:53.792924 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:06:53.793090 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:06:53.793113 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.793143 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:06:53.793170 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:06:53.795866 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:37143
I20250113 03:06:53.795918 22882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:37143 every 8 connection(s)
I20250113 03:06:53.798453 22883 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37419
I20250113 03:06:53.798497 22883 heartbeater.cc:463] Registering TS with master...
I20250113 03:06:53.798578 22883 heartbeater.cc:510] Master 127.20.18.126:37419 requested a full tablet report, sending...
I20250113 03:06:53.798763 22603 ts_manager.cc:194] Registered new tserver with Master: b1054e18ce454328aae3032e2504eaa6 (127.20.18.67:37143)
I20250113 03:06:53.799026 22603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38386
I20250113 03:06:53.799306 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003282278s
I20250113 03:06:53.799578 20553 test_util.cc:274] Using random seed: -1131814758
I20250113 03:06:53.804137 22603 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38396:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:06:53.804283 22603 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:06:53.809991 22696 tablet_service.cc:1467] Processing CreateTablet for tablet 64803ea660a04a7e841608cb6f4292cb (DEFAULT_TABLE table=test-workload [id=96e51f56f34e4bf2a306ba8383cdc3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:53.810107 22696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64803ea660a04a7e841608cb6f4292cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:53.810623 22847 tablet_service.cc:1467] Processing CreateTablet for tablet 64803ea660a04a7e841608cb6f4292cb (DEFAULT_TABLE table=test-workload [id=96e51f56f34e4bf2a306ba8383cdc3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:53.810725 22847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64803ea660a04a7e841608cb6f4292cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:53.811705 22898 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: Bootstrap starting.
I20250113 03:06:53.811744 22899 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: Bootstrap starting.
I20250113 03:06:53.811802 22772 tablet_service.cc:1467] Processing CreateTablet for tablet 64803ea660a04a7e841608cb6f4292cb (DEFAULT_TABLE table=test-workload [id=96e51f56f34e4bf2a306ba8383cdc3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:06:53.811964 22772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64803ea660a04a7e841608cb6f4292cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:06:53.812196 22898 tablet_bootstrap.cc:654] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:53.812258 22899 tablet_bootstrap.cc:654] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:53.812747 22899 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: No bootstrap required, opened a new log
I20250113 03:06:53.812772 22898 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: No bootstrap required, opened a new log
I20250113 03:06:53.812790 22899 ts_tablet_manager.cc:1397] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:53.812809 22898 ts_tablet_manager.cc:1397] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:53.812973 22898 raft_consensus.cc:357] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.812983 22899 raft_consensus.cc:357] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.813051 22898 raft_consensus.cc:383] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:53.813081 22898 raft_consensus.cc:738] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233241ab1fe34f85a6218cf643dc887a, State: Initialized, Role: FOLLOWER
I20250113 03:06:53.813060 22899 raft_consensus.cc:383] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:53.813171 22899 raft_consensus.cc:738] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1054e18ce454328aae3032e2504eaa6, State: Initialized, Role: FOLLOWER
I20250113 03:06:53.813135 22898 consensus_queue.cc:260] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.813230 22899 consensus_queue.cc:260] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.813305 22898 ts_tablet_manager.cc:1428] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:06:53.813376 22899 ts_tablet_manager.cc:1428] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:06:53.813405 22732 heartbeater.cc:502] Master 127.20.18.126:37419 was elected leader, sending a full tablet report...
I20250113 03:06:53.813489 22883 heartbeater.cc:502] Master 127.20.18.126:37419 was elected leader, sending a full tablet report...
I20250113 03:06:53.813274 22902 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: Bootstrap starting.
I20250113 03:06:53.813961 22902 tablet_bootstrap.cc:654] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: Neither blocks nor log segments found. Creating new log.
I20250113 03:06:53.814474 22902 tablet_bootstrap.cc:492] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: No bootstrap required, opened a new log
I20250113 03:06:53.814517 22902 ts_tablet_manager.cc:1397] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:06:53.814647 22902 raft_consensus.cc:357] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.814688 22902 raft_consensus.cc:383] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:06:53.814702 22902 raft_consensus.cc:738] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbe18130c5446d29cbaa535c4017c0e, State: Initialized, Role: FOLLOWER
I20250113 03:06:53.814749 22902 consensus_queue.cc:260] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.814882 22902 ts_tablet_manager.cc:1428] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:06:53.814913 22808 heartbeater.cc:502] Master 127.20.18.126:37419 was elected leader, sending a full tablet report...
I20250113 03:06:53.824082 22906 raft_consensus.cc:491] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:06:53.824148 22906 raft_consensus.cc:513] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.824270 22906 leader_election.cc:290] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1054e18ce454328aae3032e2504eaa6 (127.20.18.67:37143), 233241ab1fe34f85a6218cf643dc887a (127.20.18.65:45709)
I20250113 03:06:53.827096 22857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "4bbe18130c5446d29cbaa535c4017c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1054e18ce454328aae3032e2504eaa6" is_pre_election: true
I20250113 03:06:53.827098 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "4bbe18130c5446d29cbaa535c4017c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233241ab1fe34f85a6218cf643dc887a" is_pre_election: true
I20250113 03:06:53.827208 22857 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe18130c5446d29cbaa535c4017c0e in term 0.
I20250113 03:06:53.827210 22706 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe18130c5446d29cbaa535c4017c0e in term 0.
I20250113 03:06:53.827353 22746 leader_election.cc:304] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe18130c5446d29cbaa535c4017c0e, b1054e18ce454328aae3032e2504eaa6; no voters:
I20250113 03:06:53.827425 22906 raft_consensus.cc:2798] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:06:53.827461 22906 raft_consensus.cc:491] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:06:53.827484 22906 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:53.828017 22906 raft_consensus.cc:513] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.828101 22906 leader_election.cc:290] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [CANDIDATE]: Term 1 election: Requested vote from peers b1054e18ce454328aae3032e2504eaa6 (127.20.18.67:37143), 233241ab1fe34f85a6218cf643dc887a (127.20.18.65:45709)
I20250113 03:06:53.828230 22857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "4bbe18130c5446d29cbaa535c4017c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1054e18ce454328aae3032e2504eaa6"
I20250113 03:06:53.828290 22857 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:53.828272 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "4bbe18130c5446d29cbaa535c4017c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233241ab1fe34f85a6218cf643dc887a"
I20250113 03:06:53.828346 22706 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:06:53.828925 22857 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe18130c5446d29cbaa535c4017c0e in term 1.
I20250113 03:06:53.828935 22706 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe18130c5446d29cbaa535c4017c0e in term 1.
I20250113 03:06:53.829087 22746 leader_election.cc:304] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe18130c5446d29cbaa535c4017c0e, b1054e18ce454328aae3032e2504eaa6; no voters:
I20250113 03:06:53.829159 22906 raft_consensus.cc:2798] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:06:53.829270 22906 raft_consensus.cc:695] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 LEADER]: Becoming Leader. State: Replica: 4bbe18130c5446d29cbaa535c4017c0e, State: Running, Role: LEADER
I20250113 03:06:53.829308 22906 consensus_queue.cc:237] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:06:53.829756 22602 catalog_manager.cc:5526] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbe18130c5446d29cbaa535c4017c0e (127.20.18.66). New cstate: current_term: 1 leader_uuid: "4bbe18130c5446d29cbaa535c4017c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } health_report { overall_health: UNKNOWN } } }
I20250113 03:06:53.838810 22706 raft_consensus.cc:1270] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 1 FOLLOWER]: Refusing update from remote peer 4bbe18130c5446d29cbaa535c4017c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:53.838836 22857 raft_consensus.cc:1270] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 1 FOLLOWER]: Refusing update from remote peer 4bbe18130c5446d29cbaa535c4017c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:06:53.839002 22908 consensus_queue.cc:1035] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:06:53.839077 22906 consensus_queue.cc:1035] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:04.403452 22949 raft_consensus.cc:988] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: : Instructing follower 233241ab1fe34f85a6218cf643dc887a to start an election
I20250113 03:07:04.403522 22949 raft_consensus.cc:1076] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 LEADER]: Signalling peer 233241ab1fe34f85a6218cf643dc887a to start an election
I20250113 03:07:04.403702 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb"
dest_uuid: "233241ab1fe34f85a6218cf643dc887a"
from {username='slave'} at 127.0.0.1:57706
I20250113 03:07:04.403766 22706 raft_consensus.cc:491] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:07:04.403786 22706 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:04.404390 22706 raft_consensus.cc:513] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:07:04.404556 22706 leader_election.cc:290] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [CANDIDATE]: Term 2 election: Requested vote from peers 4bbe18130c5446d29cbaa535c4017c0e (127.20.18.66:34831), b1054e18ce454328aae3032e2504eaa6 (127.20.18.67:37143)
I20250113 03:07:04.407253 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "233241ab1fe34f85a6218cf643dc887a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15390 } } ignore_live_leader: true dest_uuid: "4bbe18130c5446d29cbaa535c4017c0e"
I20250113 03:07:04.407333 22782 raft_consensus.cc:3049] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:07:04.407349 22782 raft_consensus.cc:738] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bbe18130c5446d29cbaa535c4017c0e, State: Running, Role: LEADER
I20250113 03:07:04.407351 22857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64803ea660a04a7e841608cb6f4292cb" candidate_uuid: "233241ab1fe34f85a6218cf643dc887a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15390 } } ignore_live_leader: true dest_uuid: "b1054e18ce454328aae3032e2504eaa6"
I20250113 03:07:04.407423 22782 consensus_queue.cc:260] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15390, Committed index: 15390, Last appended: 1.15390, Last appended by leader: 15390, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:07:04.407469 22857 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:04.407500 22782 raft_consensus.cc:3054] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:04.408056 22857 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233241ab1fe34f85a6218cf643dc887a in term 2.
I20250113 03:07:04.408111 22782 raft_consensus.cc:2463] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233241ab1fe34f85a6218cf643dc887a in term 2.
I20250113 03:07:04.408236 22670 leader_election.cc:304] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233241ab1fe34f85a6218cf643dc887a, b1054e18ce454328aae3032e2504eaa6; no voters:
I20250113 03:07:04.408391 22955 raft_consensus.cc:2798] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:07:04.408501 22955 raft_consensus.cc:695] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 2 LEADER]: Becoming Leader. State: Replica: 233241ab1fe34f85a6218cf643dc887a, State: Running, Role: LEADER
I20250113 03:07:04.408566 22955 consensus_queue.cc:237] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15390, Committed index: 15390, Last appended: 1.15390, Last appended by leader: 15390, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } }
I20250113 03:07:04.408982 22602 catalog_manager.cc:5526] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a reported cstate change: term changed from 1 to 2, leader changed from 4bbe18130c5446d29cbaa535c4017c0e (127.20.18.66) to 233241ab1fe34f85a6218cf643dc887a (127.20.18.65). New cstate: current_term: 2 leader_uuid: "233241ab1fe34f85a6218cf643dc887a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233241ab1fe34f85a6218cf643dc887a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 45709 } health_report { overall_health: HEALTHY } } }
W20250113 03:07:04.416199 22963 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57440: Illegal state: Transaction hasn't been successfully started
W20250113 03:07:04.416572 22963 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57440: Illegal state: Transaction metadata for transaction 1 does not exist
I20250113 03:07:04.417104 22857 raft_consensus.cc:1270] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 2 FOLLOWER]: Refusing update from remote peer 233241ab1fe34f85a6218cf643dc887a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15390. Preceding OpId from leader: term: 2 index: 15392. (index mismatch)
I20250113 03:07:04.417120 22782 raft_consensus.cc:1270] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 2 FOLLOWER]: Refusing update from remote peer 233241ab1fe34f85a6218cf643dc887a: Log matching property violated. Preceding OpId in replica: term: 1 index: 15390. Preceding OpId from leader: term: 2 index: 15392. (index mismatch)
I20250113 03:07:04.417306 22956 consensus_queue.cc:1035] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbe18130c5446d29cbaa535c4017c0e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15391, Last known committed idx: 15390, Time since last communication: 0.000s
I20250113 03:07:04.417424 22955 consensus_queue.cc:1035] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1054e18ce454328aae3032e2504eaa6" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15391, Last known committed idx: 15390, Time since last communication: 0.000s
I20250113 03:07:04.418417 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:07:04.418514 22964 mvcc.cc:204] Tried to move back new op lower bound from 7113677309611556864 to 7113677309577969664. Current Snapshot: MvccSnapshot[applied={T|T < 7113677309611556864}]
I20250113 03:07:04.420646 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:04.420769 20553 tablet_replica.cc:331] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a: stopping tablet replica
I20250113 03:07:04.420812 20553 raft_consensus.cc:2238] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:07:04.420855 20553 raft_consensus.cc:2267] T 64803ea660a04a7e841608cb6f4292cb P 233241ab1fe34f85a6218cf643dc887a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:04.438349 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:07:04.439087 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:07:04.441432 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:04.441522 20553 tablet_replica.cc:331] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e: stopping tablet replica
I20250113 03:07:04.441574 20553 raft_consensus.cc:2238] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:04.441617 20553 raft_consensus.cc:2267] T 64803ea660a04a7e841608cb6f4292cb P 4bbe18130c5446d29cbaa535c4017c0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:04.454829 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:07:04.455545 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:07:04.458237 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:04.458338 20553 tablet_replica.cc:331] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6: stopping tablet replica
I20250113 03:07:04.458386 20553 raft_consensus.cc:2238] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:04.458441 20553 raft_consensus.cc:2267] T 64803ea660a04a7e841608cb6f4292cb P b1054e18ce454328aae3032e2504eaa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:04.480342 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:07:04.481236 20553 master.cc:537] Master@127.20.18.126:37419 shutting down...
I20250113 03:07:04.483282 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:04.483348 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:04.483397 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P c4a70472319b49338d51e96fd4bb136a: stopping tablet replica
I20250113 03:07:04.494861 20553 master.cc:559] Master@127.20.18.126:37419 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyErrorWhenNotBegun (10886 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence
I20250113 03:07:04.514442 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:38183
I20250113 03:07:04.514608 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:04.516638 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.517436 20553 server_base.cc:1034] running on GCE node
W20250113 03:07:04.517639 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.518361 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.518493 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:04.518523 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:04.518535 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737624518535 us; error 0 us; skew 500 ppm
I20250113 03:07:04.519078 20553 webserver.cc:458] Webserver started at http://127.20.18.126:44009/ using document root <none> and password file <none>
I20250113 03:07:04.519142 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:04.519188 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:04.519232 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:04.519484 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.520318 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250113 03:07:04.520754 22978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.520860 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.520907 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.520949 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:04.536949 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:04.537114 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:04.539454 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:38183
I20250113 03:07:04.541841 23030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:38183 every 8 connection(s)
I20250113 03:07:04.542068 23031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:04.544008 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d: Bootstrap starting.
I20250113 03:07:04.544678 23031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:04.545233 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d: No bootstrap required, opened a new log
I20250113 03:07:04.545333 23031 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER }
I20250113 03:07:04.545373 23031 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:04.545385 23031 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5b39b0ca8c4ae28aba3cbc68331e0d, State: Initialized, Role: FOLLOWER
I20250113 03:07:04.545424 23031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER }
I20250113 03:07:04.545461 23031 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:07:04.545490 23031 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:07:04.545509 23031 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:04.545892 23031 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER }
I20250113 03:07:04.545938 23031 leader_election.cc:304] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5b39b0ca8c4ae28aba3cbc68331e0d; no voters:
I20250113 03:07:04.546013 23031 leader_election.cc:290] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:07:04.546056 23034 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:04.546132 23031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:07:04.546193 23034 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 1 LEADER]: Becoming Leader. State: Replica: 3c5b39b0ca8c4ae28aba3cbc68331e0d, State: Running, Role: LEADER
I20250113 03:07:04.546249 23034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER }
I20250113 03:07:04.546437 23036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5b39b0ca8c4ae28aba3cbc68331e0d. Latest consensus state: current_term: 1 leader_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER } }
I20250113 03:07:04.546432 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5b39b0ca8c4ae28aba3cbc68331e0d" member_type: VOTER } }
I20250113 03:07:04.546526 23036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:04.546558 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:04.546736 23041 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:07:04.546877 23041 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:07:04.547348 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:07:04.547437 23041 catalog_manager.cc:1348] Generated new cluster ID: 9ad7bd2a5d3a4672b25d426e0c6f456f
I20250113 03:07:04.547470 23041 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:07:04.574606 23041 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:07:04.575001 23041 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:07:04.580178 23041 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d: Generated new TSK 0
I20250113 03:07:04.580247 23041 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:07:04.611147 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:04.612638 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.613510 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.614285 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.614992 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:04.615099 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:04.615135 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:04.615152 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737624615152 us; error 0 us; skew 500 ppm
I20250113 03:07:04.615809 20553 webserver.cc:458] Webserver started at http://127.20.18.65:45365/ using document root <none> and password file <none>
I20250113 03:07:04.615892 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:04.615943 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:04.616004 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:04.616273 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "a49b335430f5478eba3e5f8c6e5664ee"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.617308 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:04.617777 23060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.617970 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.618017 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "a49b335430f5478eba3e5f8c6e5664ee"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.618057 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:04.651468 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:04.651675 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:04.651809 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:04.652014 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:04.652043 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.652074 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:04.652091 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.654556 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:43949
I20250113 03:07:04.654594 23123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:43949 every 8 connection(s)
I20250113 03:07:04.654911 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:04.656232 23129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.657119 23128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.657819 23131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.658041 23124 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38183
I20250113 03:07:04.658090 23124 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:04.658187 23124 heartbeater.cc:510] Master 127.20.18.126:38183 requested a full tablet report, sending...
I20250113 03:07:04.658432 22990 ts_manager.cc:194] Registered new tserver with Master: a49b335430f5478eba3e5f8c6e5664ee (127.20.18.65:43949)
I20250113 03:07:04.658617 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:04.658747 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:04.658775 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:04.658797 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737624658797 us; error 0 us; skew 500 ppm
I20250113 03:07:04.659171 22990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34244
I20250113 03:07:04.659544 20553 webserver.cc:458] Webserver started at http://127.20.18.66:35973/ using document root <none> and password file <none>
I20250113 03:07:04.659636 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:04.659683 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:04.659730 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:04.659960 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "de73b62fcef743028788f50c4ea1ed8b"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.660840 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:04.661283 23136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.661403 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.661443 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "de73b62fcef743028788f50c4ea1ed8b"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.661484 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:04.665494 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:04.665676 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:04.665798 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:04.665958 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:04.665983 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.666010 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:04.666038 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.668625 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:39199
I20250113 03:07:04.668660 23199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:39199 every 8 connection(s)
I20250113 03:07:04.668962 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:04.670157 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:04.670992 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.671768 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:04.671813 23200 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38183
I20250113 03:07:04.671854 23200 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:04.671936 23200 heartbeater.cc:510] Master 127.20.18.126:38183 requested a full tablet report, sending...
I20250113 03:07:04.672160 22990 ts_manager.cc:194] Registered new tserver with Master: de73b62fcef743028788f50c4ea1ed8b (127.20.18.66:39199)
W20250113 03:07:04.672359 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:04.672497 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:04.672569 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:04.672597 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737624672598 us; error 0 us; skew 500 ppm
I20250113 03:07:04.672623 22990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34258
I20250113 03:07:04.673254 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43067/ using document root <none> and password file <none>
I20250113 03:07:04.673331 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:04.673377 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:04.673424 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:04.673666 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "983cf01dbcbc4dc98686de54436cd7a3"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.674443 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:04.674849 23211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.674983 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.675022 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "983cf01dbcbc4dc98686de54436cd7a3"
format_stamp: "Formatted at 2025-01-13 03:07:04 on dist-test-slave-d4dl"
I20250113 03:07:04.675065 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:04.686388 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:04.686569 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:04.686702 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:04.686882 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:04.686906 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.686934 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:04.686960 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.689992 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:44199
I20250113 03:07:04.690057 23274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:44199 every 8 connection(s)
I20250113 03:07:04.692896 23275 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38183
I20250113 03:07:04.692958 23275 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:04.693073 23275 heartbeater.cc:510] Master 127.20.18.126:38183 requested a full tablet report, sending...
I20250113 03:07:04.693325 22990 ts_manager.cc:194] Registered new tserver with Master: 983cf01dbcbc4dc98686de54436cd7a3 (127.20.18.67:44199)
I20250113 03:07:04.693478 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003313514s
I20250113 03:07:04.693718 22990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34266
I20250113 03:07:04.693811 20553 test_util.cc:274] Using random seed: -1120920526
I20250113 03:07:04.704797 22990 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34282:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:07:04.704941 22990 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:07:04.709596 23088 tablet_service.cc:1467] Processing CreateTablet for tablet 858e99a8ef624f56b4da0ccad028ea62 (DEFAULT_TABLE table=test-workload [id=5c49207eaf534285ab9a66100caeec9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:04.709735 23088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858e99a8ef624f56b4da0ccad028ea62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:04.709755 23164 tablet_service.cc:1467] Processing CreateTablet for tablet 858e99a8ef624f56b4da0ccad028ea62 (DEFAULT_TABLE table=test-workload [id=5c49207eaf534285ab9a66100caeec9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:04.709870 23164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858e99a8ef624f56b4da0ccad028ea62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:04.709985 23239 tablet_service.cc:1467] Processing CreateTablet for tablet 858e99a8ef624f56b4da0ccad028ea62 (DEFAULT_TABLE table=test-workload [id=5c49207eaf534285ab9a66100caeec9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:04.710080 23239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858e99a8ef624f56b4da0ccad028ea62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:04.710981 23290 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: Bootstrap starting.
I20250113 03:07:04.711370 23292 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: Bootstrap starting.
I20250113 03:07:04.711395 23290 tablet_bootstrap.cc:654] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:04.711480 23291 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: Bootstrap starting.
I20250113 03:07:04.711937 23292 tablet_bootstrap.cc:654] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:04.712164 23290 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: No bootstrap required, opened a new log
I20250113 03:07:04.712215 23290 ts_tablet_manager.cc:1397] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:04.712334 23290 raft_consensus.cc:357] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.712380 23290 raft_consensus.cc:383] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:04.712393 23290 raft_consensus.cc:738] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de73b62fcef743028788f50c4ea1ed8b, State: Initialized, Role: FOLLOWER
I20250113 03:07:04.712428 23290 consensus_queue.cc:260] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.712595 23290 ts_tablet_manager.cc:1428] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:04.712630 23200 heartbeater.cc:502] Master 127.20.18.126:38183 was elected leader, sending a full tablet report...
I20250113 03:07:04.712936 23291 tablet_bootstrap.cc:654] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:04.713449 23292 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: No bootstrap required, opened a new log
I20250113 03:07:04.713490 23292 ts_tablet_manager.cc:1397] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:07:04.713526 23291 tablet_bootstrap.cc:492] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: No bootstrap required, opened a new log
I20250113 03:07:04.713573 23291 ts_tablet_manager.cc:1397] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:07:04.713618 23292 raft_consensus.cc:357] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.713676 23292 raft_consensus.cc:383] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:04.713706 23292 raft_consensus.cc:738] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983cf01dbcbc4dc98686de54436cd7a3, State: Initialized, Role: FOLLOWER
I20250113 03:07:04.713721 23291 raft_consensus.cc:357] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.713752 23292 consensus_queue.cc:260] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.713778 23291 raft_consensus.cc:383] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:04.713801 23291 raft_consensus.cc:738] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49b335430f5478eba3e5f8c6e5664ee, State: Initialized, Role: FOLLOWER
I20250113 03:07:04.713850 23291 consensus_queue.cc:260] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.713898 23292 ts_tablet_manager.cc:1428] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:07:04.713997 23291 ts_tablet_manager.cc:1428] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:07:04.714018 23124 heartbeater.cc:502] Master 127.20.18.126:38183 was elected leader, sending a full tablet report...
I20250113 03:07:04.714062 23275 heartbeater.cc:502] Master 127.20.18.126:38183 was elected leader, sending a full tablet report...
I20250113 03:07:04.868993 23295 raft_consensus.cc:491] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:07:04.869102 23295 raft_consensus.cc:513] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.869313 23295 leader_election.cc:290] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a49b335430f5478eba3e5f8c6e5664ee (127.20.18.65:43949), 983cf01dbcbc4dc98686de54436cd7a3 (127.20.18.67:44199)
I20250113 03:07:04.871640 23098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "de73b62fcef743028788f50c4ea1ed8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49b335430f5478eba3e5f8c6e5664ee" is_pre_election: true
I20250113 03:07:04.871698 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "de73b62fcef743028788f50c4ea1ed8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cf01dbcbc4dc98686de54436cd7a3" is_pre_election: true
I20250113 03:07:04.871716 23098 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de73b62fcef743028788f50c4ea1ed8b in term 0.
I20250113 03:07:04.871794 23249 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de73b62fcef743028788f50c4ea1ed8b in term 0.
I20250113 03:07:04.871928 23138 leader_election.cc:304] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a49b335430f5478eba3e5f8c6e5664ee, de73b62fcef743028788f50c4ea1ed8b; no voters:
I20250113 03:07:04.872022 23295 raft_consensus.cc:2798] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:07:04.872061 23295 raft_consensus.cc:491] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:07:04.872074 23295 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:04.872685 23295 raft_consensus.cc:513] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.872792 23295 leader_election.cc:290] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [CANDIDATE]: Term 1 election: Requested vote from peers a49b335430f5478eba3e5f8c6e5664ee (127.20.18.65:43949), 983cf01dbcbc4dc98686de54436cd7a3 (127.20.18.67:44199)
I20250113 03:07:04.872931 23098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "de73b62fcef743028788f50c4ea1ed8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49b335430f5478eba3e5f8c6e5664ee"
I20250113 03:07:04.872941 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "de73b62fcef743028788f50c4ea1ed8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cf01dbcbc4dc98686de54436cd7a3"
I20250113 03:07:04.872985 23098 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:04.873003 23249 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:04.873600 23098 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de73b62fcef743028788f50c4ea1ed8b in term 1.
I20250113 03:07:04.873683 23249 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de73b62fcef743028788f50c4ea1ed8b in term 1.
I20250113 03:07:04.873781 23138 leader_election.cc:304] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a49b335430f5478eba3e5f8c6e5664ee, de73b62fcef743028788f50c4ea1ed8b; no voters:
I20250113 03:07:04.873871 23295 raft_consensus.cc:2798] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:04.874014 23295 raft_consensus.cc:695] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 LEADER]: Becoming Leader. State: Replica: de73b62fcef743028788f50c4ea1ed8b, State: Running, Role: LEADER
I20250113 03:07:04.874065 23295 consensus_queue.cc:237] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:04.874518 22990 catalog_manager.cc:5526] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b reported cstate change: term changed from 0 to 1, leader changed from <none> to de73b62fcef743028788f50c4ea1ed8b (127.20.18.66). New cstate: current_term: 1 leader_uuid: "de73b62fcef743028788f50c4ea1ed8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:04.896062 23249 raft_consensus.cc:1270] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 1 FOLLOWER]: Refusing update from remote peer de73b62fcef743028788f50c4ea1ed8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:04.896299 23300 consensus_queue.cc:1035] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:04.896306 23098 raft_consensus.cc:1270] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 1 FOLLOWER]: Refusing update from remote peer de73b62fcef743028788f50c4ea1ed8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:04.896556 23300 consensus_queue.cc:1035] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:15.406242 23336 raft_consensus.cc:988] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: : Instructing follower a49b335430f5478eba3e5f8c6e5664ee to start an election
I20250113 03:07:15.406315 23336 raft_consensus.cc:1076] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 LEADER]: Signalling peer a49b335430f5478eba3e5f8c6e5664ee to start an election
I20250113 03:07:15.406488 23098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62"
dest_uuid: "a49b335430f5478eba3e5f8c6e5664ee"
from {username='slave'} at 127.0.0.1:50096
I20250113 03:07:15.406572 23098 raft_consensus.cc:491] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:07:15.406601 23098 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:15.407212 23098 raft_consensus.cc:513] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:15.407367 23098 leader_election.cc:290] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [CANDIDATE]: Term 2 election: Requested vote from peers de73b62fcef743028788f50c4ea1ed8b (127.20.18.66:39199), 983cf01dbcbc4dc98686de54436cd7a3 (127.20.18.67:44199)
I20250113 03:07:15.410741 23174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "a49b335430f5478eba3e5f8c6e5664ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 15673 } } ignore_live_leader: true dest_uuid: "de73b62fcef743028788f50c4ea1ed8b"
I20250113 03:07:15.410741 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e99a8ef624f56b4da0ccad028ea62" candidate_uuid: "a49b335430f5478eba3e5f8c6e5664ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 15673 } } ignore_live_leader: true dest_uuid: "983cf01dbcbc4dc98686de54436cd7a3"
I20250113 03:07:15.410856 23174 raft_consensus.cc:3049] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:07:15.410899 23174 raft_consensus.cc:738] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de73b62fcef743028788f50c4ea1ed8b, State: Running, Role: LEADER
I20250113 03:07:15.410862 23249 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:15.410954 23174 consensus_queue.cc:260] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15673, Committed index: 15673, Last appended: 1.15673, Last appended by leader: 15673, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:15.411052 23174 raft_consensus.cc:3054] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:15.411681 23249 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a49b335430f5478eba3e5f8c6e5664ee in term 2.
I20250113 03:07:15.411681 23174 raft_consensus.cc:2463] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a49b335430f5478eba3e5f8c6e5664ee in term 2.
I20250113 03:07:15.411890 23064 leader_election.cc:304] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 983cf01dbcbc4dc98686de54436cd7a3, a49b335430f5478eba3e5f8c6e5664ee; no voters:
I20250113 03:07:15.412068 23354 raft_consensus.cc:2798] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:07:15.412178 23354 raft_consensus.cc:695] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 2 LEADER]: Becoming Leader. State: Replica: a49b335430f5478eba3e5f8c6e5664ee, State: Running, Role: LEADER
I20250113 03:07:15.412235 23354 consensus_queue.cc:237] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15673, Committed index: 15673, Last appended: 1.15673, Last appended by leader: 15673, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } }
I20250113 03:07:15.412688 22992 catalog_manager.cc:5526] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee reported cstate change: term changed from 1 to 2, leader changed from de73b62fcef743028788f50c4ea1ed8b (127.20.18.66) to a49b335430f5478eba3e5f8c6e5664ee (127.20.18.65). New cstate: current_term: 2 leader_uuid: "a49b335430f5478eba3e5f8c6e5664ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a49b335430f5478eba3e5f8c6e5664ee" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:15.418686 23174 raft_consensus.cc:1270] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 2 FOLLOWER]: Refusing update from remote peer a49b335430f5478eba3e5f8c6e5664ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 15673. Preceding OpId from leader: term: 2 index: 15675. (index mismatch)
I20250113 03:07:15.418686 23249 raft_consensus.cc:1270] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 2 FOLLOWER]: Refusing update from remote peer a49b335430f5478eba3e5f8c6e5664ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 15673. Preceding OpId from leader: term: 2 index: 15675. (index mismatch)
I20250113 03:07:15.418851 23354 consensus_queue.cc:1035] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "983cf01dbcbc4dc98686de54436cd7a3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15674, Last known committed idx: 15673, Time since last communication: 0.000s
I20250113 03:07:15.418937 23355 consensus_queue.cc:1035] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "de73b62fcef743028788f50c4ea1ed8b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15674, Last known committed idx: 15673, Time since last communication: 0.000s
W20250113 03:07:15.420400 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction 0 already open
W20250113 03:07:15.420801 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Cannot finalize transaction in state: OPEN
W20250113 03:07:15.421830 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction 0 commit already in progress
W20250113 03:07:15.422152 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:07:15.423349 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction 0 commit already in progress
W20250113 03:07:15.423681 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction metadata for transaction 0 already set
W20250113 03:07:15.423938 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:07:15.424175 23362 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49280: Illegal state: Transaction metadata for transaction 0 already exists
I20250113 03:07:15.424342 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:07:15.426358 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:15.426440 20553 tablet_replica.cc:331] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee: stopping tablet replica
I20250113 03:07:15.426483 20553 raft_consensus.cc:2238] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:07:15.426532 20553 raft_consensus.cc:2267] T 858e99a8ef624f56b4da0ccad028ea62 P a49b335430f5478eba3e5f8c6e5664ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:15.431492 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:07:15.432211 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:07:15.434556 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:15.434641 20553 tablet_replica.cc:331] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b: stopping tablet replica
I20250113 03:07:15.434689 20553 raft_consensus.cc:2238] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:15.434724 20553 raft_consensus.cc:2267] T 858e99a8ef624f56b4da0ccad028ea62 P de73b62fcef743028788f50c4ea1ed8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:15.437919 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:07:15.438609 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:07:15.441104 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:15.441193 20553 tablet_replica.cc:331] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3: stopping tablet replica
I20250113 03:07:15.441232 20553 raft_consensus.cc:2238] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:15.441277 20553 raft_consensus.cc:2267] T 858e99a8ef624f56b4da0ccad028ea62 P 983cf01dbcbc4dc98686de54436cd7a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:15.463341 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:07:15.464100 20553 master.cc:537] Master@127.20.18.126:38183 shutting down...
I20250113 03:07:15.466027 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:15.466087 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:15.466114 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3c5b39b0ca8c4ae28aba3cbc68331e0d: stopping tablet replica
I20250113 03:07:15.477615 20553 master.cc:559] Master@127.20.18.126:38183 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence (10981 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence
I20250113 03:07:15.496217 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:37481
I20250113 03:07:15.496383 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:15.497648 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.498593 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.499325 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.500032 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:15.500123 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:15.500147 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:15.500170 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737635500169 us; error 0 us; skew 500 ppm
I20250113 03:07:15.500787 20553 webserver.cc:458] Webserver started at http://127.20.18.126:34707/ using document root <none> and password file <none>
I20250113 03:07:15.500862 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:15.500905 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:15.500952 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:15.501199 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "1addbfad2772481f992cb2a2c17f081f"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.501976 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:15.502369 23378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.502483 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.502524 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "1addbfad2772481f992cb2a2c17f081f"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.502575 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:15.526060 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:15.526234 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:15.528460 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:37481
I20250113 03:07:15.530877 23430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:37481 every 8 connection(s)
I20250113 03:07:15.535488 23431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:15.536437 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f: Bootstrap starting.
I20250113 03:07:15.536783 23431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:15.537204 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f: No bootstrap required, opened a new log
I20250113 03:07:15.537297 23431 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER }
I20250113 03:07:15.537339 23431 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:15.537353 23431 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1addbfad2772481f992cb2a2c17f081f, State: Initialized, Role: FOLLOWER
I20250113 03:07:15.537389 23431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER }
I20250113 03:07:15.537429 23431 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:07:15.537461 23431 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:07:15.537482 23431 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:15.537916 23431 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER }
I20250113 03:07:15.537963 23431 leader_election.cc:304] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1addbfad2772481f992cb2a2c17f081f; no voters:
I20250113 03:07:15.538033 23431 leader_election.cc:290] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:07:15.538093 23434 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:15.538141 23431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:07:15.538233 23434 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 1 LEADER]: Becoming Leader. State: Replica: 1addbfad2772481f992cb2a2c17f081f, State: Running, Role: LEADER
I20250113 03:07:15.538311 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER }
I20250113 03:07:15.538524 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1addbfad2772481f992cb2a2c17f081f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER } }
I20250113 03:07:15.538565 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1addbfad2772481f992cb2a2c17f081f. Latest consensus state: current_term: 1 leader_uuid: "1addbfad2772481f992cb2a2c17f081f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1addbfad2772481f992cb2a2c17f081f" member_type: VOTER } }
I20250113 03:07:15.538674 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:15.538707 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:15.538872 23441 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:07:15.539012 23441 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:07:15.539506 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:07:15.539513 23441 catalog_manager.cc:1348] Generated new cluster ID: 0e4e341ee4b24432849ca65a4cf2a0c2
I20250113 03:07:15.539582 23441 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:07:15.545038 23441 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:07:15.545373 23441 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:07:15.550135 23441 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f: Generated new TSK 0
I20250113 03:07:15.550194 23441 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:07:15.555193 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:15.557118 23453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.557304 23452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.558171 23455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.558872 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:15.558969 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:15.559003 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:15.559026 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737635559025 us; error 0 us; skew 500 ppm
I20250113 03:07:15.559651 20553 webserver.cc:458] Webserver started at http://127.20.18.65:42887/ using document root <none> and password file <none>
I20250113 03:07:15.559732 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:15.559773 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:15.559819 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:15.560060 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "55d86e2500ba4994ae2e50c44b844836"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.560925 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:15.561347 23460 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.561479 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.561520 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "55d86e2500ba4994ae2e50c44b844836"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.561570 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:15.571177 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:15.571352 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:15.571466 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:15.571630 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:15.571655 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.571683 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:15.571709 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.574280 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:33825
I20250113 03:07:15.574476 23523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:33825 every 8 connection(s)
I20250113 03:07:15.574725 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:15.575907 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.576736 23529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.577505 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:15.577770 23524 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37481
I20250113 03:07:15.577826 23524 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:15.577926 23524 heartbeater.cc:510] Master 127.20.18.126:37481 requested a full tablet report, sending...
W20250113 03:07:15.578135 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.578168 23395 ts_manager.cc:194] Registered new tserver with Master: 55d86e2500ba4994ae2e50c44b844836 (127.20.18.65:33825)
I20250113 03:07:15.578291 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:15.578323 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:15.578334 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737635578335 us; error 0 us; skew 500 ppm
I20250113 03:07:15.578698 23395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33504
I20250113 03:07:15.579013 20553 webserver.cc:458] Webserver started at http://127.20.18.66:44237/ using document root <none> and password file <none>
I20250113 03:07:15.579106 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:15.579150 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:15.579195 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:15.579420 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "eca07fb93ba243b18c281453d97422f7"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.580143 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:15.580494 23536 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.580641 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.580682 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "eca07fb93ba243b18c281453d97422f7"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.580708 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:15.591533 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:15.591692 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:15.591820 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:15.591989 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:15.592015 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.592043 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:15.592069 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.594779 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:38405
I20250113 03:07:15.594826 23599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:38405 every 8 connection(s)
I20250113 03:07:15.595113 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:15.596969 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:15.597189 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.597931 23600 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37481
W20250113 03:07:15.597952 23606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:15.597990 23600 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:15.598146 23600 heartbeater.cc:510] Master 127.20.18.126:37481 requested a full tablet report, sending...
I20250113 03:07:15.598356 23395 ts_manager.cc:194] Registered new tserver with Master: eca07fb93ba243b18c281453d97422f7 (127.20.18.66:38405)
I20250113 03:07:15.598654 23395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33510
I20250113 03:07:15.598695 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:15.598852 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:15.598886 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:15.598906 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737635598905 us; error 0 us; skew 500 ppm
I20250113 03:07:15.599413 20553 webserver.cc:458] Webserver started at http://127.20.18.67:44993/ using document root <none> and password file <none>
I20250113 03:07:15.599483 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:15.599529 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:15.599571 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:15.599814 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "522b8c718b434c17bfc95a54f5c41de0"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.600632 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:15.601032 23611 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.601141 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.601179 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "522b8c718b434c17bfc95a54f5c41de0"
format_stamp: "Formatted at 2025-01-13 03:07:15 on dist-test-slave-d4dl"
I20250113 03:07:15.601222 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:15.610460 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:15.610641 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:15.610774 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:15.610927 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:15.610951 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.610980 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:15.611006 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.613862 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:39833
I20250113 03:07:15.613915 23674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:39833 every 8 connection(s)
I20250113 03:07:15.616397 23675 heartbeater.cc:346] Connected to a master server at 127.20.18.126:37481
I20250113 03:07:15.616443 23675 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:15.616605 23675 heartbeater.cc:510] Master 127.20.18.126:37481 requested a full tablet report, sending...
I20250113 03:07:15.616817 23395 ts_manager.cc:194] Registered new tserver with Master: 522b8c718b434c17bfc95a54f5c41de0 (127.20.18.67:39833)
I20250113 03:07:15.617105 23395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33524
I20250113 03:07:15.617292 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003260614s
I20250113 03:07:15.617569 20553 test_util.cc:274] Using random seed: -1109996767
I20250113 03:07:15.621414 23395 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33530:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:07:15.621528 23395 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:07:15.625797 23488 tablet_service.cc:1467] Processing CreateTablet for tablet b62cfbb1c6e54c9abab4e5428091e9a9 (DEFAULT_TABLE table=test-workload [id=65290848c79543a793cf3092e441feaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:15.625916 23488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62cfbb1c6e54c9abab4e5428091e9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:15.625917 23564 tablet_service.cc:1467] Processing CreateTablet for tablet b62cfbb1c6e54c9abab4e5428091e9a9 (DEFAULT_TABLE table=test-workload [id=65290848c79543a793cf3092e441feaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:15.626060 23564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62cfbb1c6e54c9abab4e5428091e9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:15.626346 23639 tablet_service.cc:1467] Processing CreateTablet for tablet b62cfbb1c6e54c9abab4e5428091e9a9 (DEFAULT_TABLE table=test-workload [id=65290848c79543a793cf3092e441feaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:15.626446 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62cfbb1c6e54c9abab4e5428091e9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:15.627125 23690 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: Bootstrap starting.
I20250113 03:07:15.627337 23691 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: Bootstrap starting.
I20250113 03:07:15.627691 23692 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: Bootstrap starting.
I20250113 03:07:15.627784 23690 tablet_bootstrap.cc:654] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:15.627898 23691 tablet_bootstrap.cc:654] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:15.628144 23692 tablet_bootstrap.cc:654] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:15.628576 23691 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: No bootstrap required, opened a new log
I20250113 03:07:15.628628 23691 ts_tablet_manager.cc:1397] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:15.628719 23690 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: No bootstrap required, opened a new log
I20250113 03:07:15.628755 23690 ts_tablet_manager.cc:1397] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:07:15.628772 23691 raft_consensus.cc:357] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.628845 23691 raft_consensus.cc:383] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:15.628871 23691 raft_consensus.cc:738] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d86e2500ba4994ae2e50c44b844836, State: Initialized, Role: FOLLOWER
I20250113 03:07:15.628878 23690 raft_consensus.cc:357] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.628940 23690 raft_consensus.cc:383] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:15.628966 23690 raft_consensus.cc:738] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca07fb93ba243b18c281453d97422f7, State: Initialized, Role: FOLLOWER
I20250113 03:07:15.628976 23691 consensus_queue.cc:260] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.629005 23690 consensus_queue.cc:260] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.629166 23691 ts_tablet_manager.cc:1428] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 03:07:15.629230 23600 heartbeater.cc:502] Master 127.20.18.126:37481 was elected leader, sending a full tablet report...
I20250113 03:07:15.629189 23690 ts_tablet_manager.cc:1428] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:07:15.629169 23692 tablet_bootstrap.cc:492] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: No bootstrap required, opened a new log
I20250113 03:07:15.629338 23692 ts_tablet_manager.cc:1397] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:07:15.629459 23692 raft_consensus.cc:357] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.629505 23692 raft_consensus.cc:383] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:15.629524 23692 raft_consensus.cc:738] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522b8c718b434c17bfc95a54f5c41de0, State: Initialized, Role: FOLLOWER
I20250113 03:07:15.629580 23692 consensus_queue.cc:260] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.629739 23675 heartbeater.cc:502] Master 127.20.18.126:37481 was elected leader, sending a full tablet report...
I20250113 03:07:15.629745 23692 ts_tablet_manager.cc:1428] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:15.629189 23524 heartbeater.cc:502] Master 127.20.18.126:37481 was elected leader, sending a full tablet report...
I20250113 03:07:15.639079 23696 raft_consensus.cc:491] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:07:15.639132 23696 raft_consensus.cc:513] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.639278 23696 leader_election.cc:290] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522b8c718b434c17bfc95a54f5c41de0 (127.20.18.67:39833), eca07fb93ba243b18c281453d97422f7 (127.20.18.66:38405)
I20250113 03:07:15.641397 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cfbb1c6e54c9abab4e5428091e9a9" candidate_uuid: "55d86e2500ba4994ae2e50c44b844836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522b8c718b434c17bfc95a54f5c41de0" is_pre_election: true
I20250113 03:07:15.641482 23574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cfbb1c6e54c9abab4e5428091e9a9" candidate_uuid: "55d86e2500ba4994ae2e50c44b844836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca07fb93ba243b18c281453d97422f7" is_pre_election: true
I20250113 03:07:15.641481 23649 raft_consensus.cc:2463] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d86e2500ba4994ae2e50c44b844836 in term 0.
I20250113 03:07:15.641564 23574 raft_consensus.cc:2463] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d86e2500ba4994ae2e50c44b844836 in term 0.
I20250113 03:07:15.641713 23462 leader_election.cc:304] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522b8c718b434c17bfc95a54f5c41de0, 55d86e2500ba4994ae2e50c44b844836; no voters:
I20250113 03:07:15.641810 23696 raft_consensus.cc:2798] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:07:15.641847 23696 raft_consensus.cc:491] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:07:15.641866 23696 raft_consensus.cc:3054] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:15.642304 23696 raft_consensus.cc:513] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.642400 23696 leader_election.cc:290] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [CANDIDATE]: Term 1 election: Requested vote from peers 522b8c718b434c17bfc95a54f5c41de0 (127.20.18.67:39833), eca07fb93ba243b18c281453d97422f7 (127.20.18.66:38405)
I20250113 03:07:15.642534 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cfbb1c6e54c9abab4e5428091e9a9" candidate_uuid: "55d86e2500ba4994ae2e50c44b844836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522b8c718b434c17bfc95a54f5c41de0"
I20250113 03:07:15.642534 23574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cfbb1c6e54c9abab4e5428091e9a9" candidate_uuid: "55d86e2500ba4994ae2e50c44b844836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca07fb93ba243b18c281453d97422f7"
I20250113 03:07:15.642629 23649 raft_consensus.cc:3054] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:15.642634 23574 raft_consensus.cc:3054] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:15.643270 23574 raft_consensus.cc:2463] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d86e2500ba4994ae2e50c44b844836 in term 1.
I20250113 03:07:15.643270 23649 raft_consensus.cc:2463] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d86e2500ba4994ae2e50c44b844836 in term 1.
I20250113 03:07:15.643461 23464 leader_election.cc:304] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d86e2500ba4994ae2e50c44b844836, eca07fb93ba243b18c281453d97422f7; no voters:
I20250113 03:07:15.643569 23696 raft_consensus.cc:2798] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:15.643697 23696 raft_consensus.cc:695] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 1 LEADER]: Becoming Leader. State: Replica: 55d86e2500ba4994ae2e50c44b844836, State: Running, Role: LEADER
I20250113 03:07:15.643744 23696 consensus_queue.cc:237] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } }
I20250113 03:07:15.644183 23395 catalog_manager.cc:5526] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d86e2500ba4994ae2e50c44b844836 (127.20.18.65). New cstate: current_term: 1 leader_uuid: "55d86e2500ba4994ae2e50c44b844836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55d86e2500ba4994ae2e50c44b844836" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:15.660504 23574 raft_consensus.cc:1270] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 1 FOLLOWER]: Refusing update from remote peer 55d86e2500ba4994ae2e50c44b844836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:15.660518 23649 raft_consensus.cc:1270] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 1 FOLLOWER]: Refusing update from remote peer 55d86e2500ba4994ae2e50c44b844836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:15.660730 23700 consensus_queue.cc:1035] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522b8c718b434c17bfc95a54f5c41de0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:15.660835 23696 consensus_queue.cc:1035] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca07fb93ba243b18c281453d97422f7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 03:07:25.658560 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Transaction 0 already open
W20250113 03:07:25.658859 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Cannot finalize transaction in state: OPEN
W20250113 03:07:25.660117 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Transaction metadata for transaction 0 already set
W20250113 03:07:25.660362 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:07:25.660610 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:07:25.660863 23707 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37330: Illegal state: Cannot begin committing transaction in state: INITIALIZING
I20250113 03:07:25.661008 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:07:25.663224 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:25.663308 20553 tablet_replica.cc:331] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836: stopping tablet replica
I20250113 03:07:25.663362 20553 raft_consensus.cc:2238] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:25.663404 20553 raft_consensus.cc:2267] T b62cfbb1c6e54c9abab4e5428091e9a9 P 55d86e2500ba4994ae2e50c44b844836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:25.676921 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:07:25.677599 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:07:25.680298 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:25.680388 20553 tablet_replica.cc:331] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7: stopping tablet replica
I20250113 03:07:25.680428 20553 raft_consensus.cc:2238] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:25.680464 20553 raft_consensus.cc:2267] T b62cfbb1c6e54c9abab4e5428091e9a9 P eca07fb93ba243b18c281453d97422f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:25.701429 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:07:25.702183 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:07:25.704442 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:25.704547 20553 tablet_replica.cc:331] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0: stopping tablet replica
I20250113 03:07:25.704602 20553 raft_consensus.cc:2238] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:25.704636 20553 raft_consensus.cc:2267] T b62cfbb1c6e54c9abab4e5428091e9a9 P 522b8c718b434c17bfc95a54f5c41de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:25.717689 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:07:25.718437 20553 master.cc:537] Master@127.20.18.126:37481 shutting down...
I20250113 03:07:25.720359 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:25.720409 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:25.720436 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1addbfad2772481f992cb2a2c17f081f: stopping tablet replica
I20250113 03:07:25.732041 20553 master.cc:559] Master@127.20.18.126:37481 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence (10253 ms)
[ RUN ] TxnParticipantITest.TestProxyNonLeader
I20250113 03:07:25.750002 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:42883
I20250113 03:07:25.750141 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:25.752388 23754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.753154 23751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:25.753437 20553 server_base.cc:1034] running on GCE node
W20250113 03:07:25.754068 23752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:25.754204 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:25.754235 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:25.754245 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737645754246 us; error 0 us; skew 500 ppm
I20250113 03:07:25.754822 20553 webserver.cc:458] Webserver started at http://127.20.18.126:41175/ using document root <none> and password file <none>
I20250113 03:07:25.754890 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:25.754917 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:25.754946 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:25.755177 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "b9f190c122114178aa03b0a20094b8ca"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.755924 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:25.756278 23759 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.756386 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.756440 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "b9f190c122114178aa03b0a20094b8ca"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.756496 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:25.767840 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:25.767972 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:25.770229 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:42883
I20250113 03:07:25.774497 23812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:25.780265 23811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:42883 every 8 connection(s)
I20250113 03:07:25.781895 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca: Bootstrap starting.
I20250113 03:07:25.782145 23812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:25.782558 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca: No bootstrap required, opened a new log
I20250113 03:07:25.782652 23812 raft_consensus.cc:357] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER }
I20250113 03:07:25.782689 23812 raft_consensus.cc:383] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:25.782702 23812 raft_consensus.cc:738] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f190c122114178aa03b0a20094b8ca, State: Initialized, Role: FOLLOWER
I20250113 03:07:25.782738 23812 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER }
I20250113 03:07:25.782776 23812 raft_consensus.cc:397] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:07:25.782801 23812 raft_consensus.cc:491] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:07:25.782819 23812 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:25.783200 23812 raft_consensus.cc:513] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER }
I20250113 03:07:25.783246 23812 leader_election.cc:304] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f190c122114178aa03b0a20094b8ca; no voters:
I20250113 03:07:25.783324 23812 leader_election.cc:290] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:07:25.783386 23815 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:25.783463 23812 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:07:25.783492 23815 raft_consensus.cc:695] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 1 LEADER]: Becoming Leader. State: Replica: b9f190c122114178aa03b0a20094b8ca, State: Running, Role: LEADER
I20250113 03:07:25.783592 23815 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER }
I20250113 03:07:25.783789 23816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f190c122114178aa03b0a20094b8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER } }
I20250113 03:07:25.783793 23817 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f190c122114178aa03b0a20094b8ca. Latest consensus state: current_term: 1 leader_uuid: "b9f190c122114178aa03b0a20094b8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f190c122114178aa03b0a20094b8ca" member_type: VOTER } }
I20250113 03:07:25.783923 23816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:25.784123 23817 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:25.784147 23825 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:07:25.784323 23825 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:07:25.784680 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:07:25.784809 23825 catalog_manager.cc:1348] Generated new cluster ID: 680768ca6ddc4599a6655b8dc34b5056
I20250113 03:07:25.784852 23825 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:07:25.791993 23825 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:07:25.792302 23825 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:07:25.798872 23825 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca: Generated new TSK 0
I20250113 03:07:25.798930 23825 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:07:25.800174 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:25.801301 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.802165 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.802909 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:25.803591 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:25.803686 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:25.803716 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:25.803730 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737645803730 us; error 0 us; skew 500 ppm
I20250113 03:07:25.804291 20553 webserver.cc:458] Webserver started at http://127.20.18.65:42403/ using document root <none> and password file <none>
I20250113 03:07:25.804354 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:25.804394 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:25.804433 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:25.804685 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "5a8134bb26634adc9cbe9846b0bde564"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.805529 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:25.806007 23841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.806144 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.806190 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "5a8134bb26634adc9cbe9846b0bde564"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.806228 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:25.822077 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:25.822225 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:25.822347 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:25.822506 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:25.822530 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.822557 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:25.822582 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.825069 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:36179
I20250113 03:07:25.825542 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:07:25.825706 23904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:36179 every 8 connection(s)
W20250113 03:07:25.826731 23907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.827508 23908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.828176 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:25.828830 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:25.828866 23905 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42883
I20250113 03:07:25.828940 23905 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:25.828970 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:25.829021 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:25.829046 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737645829046 us; error 0 us; skew 500 ppm
I20250113 03:07:25.829051 23905 heartbeater.cc:510] Master 127.20.18.126:42883 requested a full tablet report, sending...
I20250113 03:07:25.829315 23776 ts_manager.cc:194] Registered new tserver with Master: 5a8134bb26634adc9cbe9846b0bde564 (127.20.18.65:36179)
I20250113 03:07:25.829757 20553 webserver.cc:458] Webserver started at http://127.20.18.66:39317/ using document root <none> and password file <none>
I20250113 03:07:25.829836 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:25.829881 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:25.829923 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:25.830082 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44498
I20250113 03:07:25.830192 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "2b094dbd07a141a58373d6b1085aad61"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.831053 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:25.831436 23917 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.831554 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.831593 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "2b094dbd07a141a58373d6b1085aad61"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.831637 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:25.838311 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:25.838444 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:25.838551 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:25.838696 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:25.838718 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.838766 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:25.838788 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.841488 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:35965
I20250113 03:07:25.841533 23980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:35965 every 8 connection(s)
I20250113 03:07:25.841802 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:25.843477 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.843616 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:25.844249 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:25.844270 23981 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42883
I20250113 03:07:25.844331 23981 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:25.844425 23981 heartbeater.cc:510] Master 127.20.18.126:42883 requested a full tablet report, sending...
I20250113 03:07:25.844663 23776 ts_manager.cc:194] Registered new tserver with Master: 2b094dbd07a141a58373d6b1085aad61 (127.20.18.66:35965)
I20250113 03:07:25.844880 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:25.844985 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:25.845013 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:25.845021 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44514
I20250113 03:07:25.845036 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737645845037 us; error 0 us; skew 500 ppm
I20250113 03:07:25.845660 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43273/ using document root <none> and password file <none>
I20250113 03:07:25.845726 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:25.845755 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:25.845784 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:25.845973 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "4c483e6dd58240a499832e28c90ef370"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.846652 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:25.846978 23992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.847071 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.847108 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "4c483e6dd58240a499832e28c90ef370"
format_stamp: "Formatted at 2025-01-13 03:07:25 on dist-test-slave-d4dl"
I20250113 03:07:25.847147 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:25.855960 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:25.856139 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:25.856268 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:25.856432 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:25.856456 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.856484 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:25.856511 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.859614 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34307
I20250113 03:07:25.859664 24055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34307 every 8 connection(s)
I20250113 03:07:25.862149 24056 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42883
I20250113 03:07:25.862200 24056 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:25.862294 24056 heartbeater.cc:510] Master 127.20.18.126:42883 requested a full tablet report, sending...
I20250113 03:07:25.862498 23776 ts_manager.cc:194] Registered new tserver with Master: 4c483e6dd58240a499832e28c90ef370 (127.20.18.67:34307)
I20250113 03:07:25.862802 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44526
I20250113 03:07:25.863075 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003272965s
I20250113 03:07:25.863361 20553 test_util.cc:274] Using random seed: -1099750975
I20250113 03:07:25.867349 23776 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44528:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:07:25.867460 23776 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:07:25.871600 23869 tablet_service.cc:1467] Processing CreateTablet for tablet 8aadaee7eab64b119a62c6b3e8d9ef23 (DEFAULT_TABLE table=test-workload [id=9b2dab3071bd453999eee9865731a775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:25.871727 23869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aadaee7eab64b119a62c6b3e8d9ef23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:25.871774 24020 tablet_service.cc:1467] Processing CreateTablet for tablet 8aadaee7eab64b119a62c6b3e8d9ef23 (DEFAULT_TABLE table=test-workload [id=9b2dab3071bd453999eee9865731a775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:25.871871 24020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aadaee7eab64b119a62c6b3e8d9ef23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:25.872138 23945 tablet_service.cc:1467] Processing CreateTablet for tablet 8aadaee7eab64b119a62c6b3e8d9ef23 (DEFAULT_TABLE table=test-workload [id=9b2dab3071bd453999eee9865731a775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:25.872231 23945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aadaee7eab64b119a62c6b3e8d9ef23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:25.873073 24071 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: Bootstrap starting.
I20250113 03:07:25.873205 24072 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Bootstrap starting.
I20250113 03:07:25.873471 24073 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: Bootstrap starting.
I20250113 03:07:25.873535 24071 tablet_bootstrap.cc:654] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:25.873636 24072 tablet_bootstrap.cc:654] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:25.873891 24073 tablet_bootstrap.cc:654] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:25.874202 24071 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: No bootstrap required, opened a new log
I20250113 03:07:25.874251 24071 ts_tablet_manager.cc:1397] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:25.874354 24072 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: No bootstrap required, opened a new log
I20250113 03:07:25.874421 24072 ts_tablet_manager.cc:1397] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:25.874375 24071 raft_consensus.cc:357] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.874454 24071 raft_consensus.cc:383] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:25.874483 24071 raft_consensus.cc:738] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c483e6dd58240a499832e28c90ef370, State: Initialized, Role: FOLLOWER
I20250113 03:07:25.874533 24071 consensus_queue.cc:260] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.874557 24072 raft_consensus.cc:357] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.874608 24072 raft_consensus.cc:383] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:25.874636 24072 raft_consensus.cc:738] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8134bb26634adc9cbe9846b0bde564, State: Initialized, Role: FOLLOWER
I20250113 03:07:25.874673 24071 ts_tablet_manager.cc:1428] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:25.874698 24056 heartbeater.cc:502] Master 127.20.18.126:42883 was elected leader, sending a full tablet report...
I20250113 03:07:25.874354 24073 tablet_bootstrap.cc:492] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: No bootstrap required, opened a new log
I20250113 03:07:25.874794 24073 ts_tablet_manager.cc:1397] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:25.874773 24072 consensus_queue.cc:260] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.874944 24072 ts_tablet_manager.cc:1428] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:07:25.874971 23905 heartbeater.cc:502] Master 127.20.18.126:42883 was elected leader, sending a full tablet report...
I20250113 03:07:25.874924 24073 raft_consensus.cc:357] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.875022 24073 raft_consensus.cc:383] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:25.875043 24073 raft_consensus.cc:738] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b094dbd07a141a58373d6b1085aad61, State: Initialized, Role: FOLLOWER
I20250113 03:07:25.875147 24073 consensus_queue.cc:260] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:25.875344 24073 ts_tablet_manager.cc:1428] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:07:25.875418 23981 heartbeater.cc:502] Master 127.20.18.126:42883 was elected leader, sending a full tablet report...
W20250113 03:07:26.076074 23906 tablet.cc:2367] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 03:07:26.177278 24077 raft_consensus.cc:491] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:07:26.177376 24077 raft_consensus.cc:513] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:26.177567 24077 leader_election.cc:290] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8134bb26634adc9cbe9846b0bde564 (127.20.18.65:36179), 2b094dbd07a141a58373d6b1085aad61 (127.20.18.66:35965)
I20250113 03:07:26.179867 23879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "4c483e6dd58240a499832e28c90ef370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8134bb26634adc9cbe9846b0bde564" is_pre_election: true
I20250113 03:07:26.179944 23879 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c483e6dd58240a499832e28c90ef370 in term 0.
I20250113 03:07:26.179894 23955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "4c483e6dd58240a499832e28c90ef370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b094dbd07a141a58373d6b1085aad61" is_pre_election: true
I20250113 03:07:26.179997 23955 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c483e6dd58240a499832e28c90ef370 in term 0.
I20250113 03:07:26.180083 23996 leader_election.cc:304] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c483e6dd58240a499832e28c90ef370, 5a8134bb26634adc9cbe9846b0bde564; no voters:
I20250113 03:07:26.180161 24077 raft_consensus.cc:2798] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:07:26.180199 24077 raft_consensus.cc:491] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:07:26.180217 24077 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:26.180794 24077 raft_consensus.cc:513] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:26.180884 24077 leader_election.cc:290] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [CANDIDATE]: Term 1 election: Requested vote from peers 5a8134bb26634adc9cbe9846b0bde564 (127.20.18.65:36179), 2b094dbd07a141a58373d6b1085aad61 (127.20.18.66:35965)
I20250113 03:07:26.181020 23879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "4c483e6dd58240a499832e28c90ef370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8134bb26634adc9cbe9846b0bde564"
I20250113 03:07:26.181061 23955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "4c483e6dd58240a499832e28c90ef370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b094dbd07a141a58373d6b1085aad61"
I20250113 03:07:26.181083 23879 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:26.181097 23955 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:26.181656 23955 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c483e6dd58240a499832e28c90ef370 in term 1.
I20250113 03:07:26.181660 23879 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c483e6dd58240a499832e28c90ef370 in term 1.
I20250113 03:07:26.181816 23996 leader_election.cc:304] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b094dbd07a141a58373d6b1085aad61, 4c483e6dd58240a499832e28c90ef370; no voters:
I20250113 03:07:26.181893 24077 raft_consensus.cc:2798] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:26.182018 24077 raft_consensus.cc:695] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 LEADER]: Becoming Leader. State: Replica: 4c483e6dd58240a499832e28c90ef370, State: Running, Role: LEADER
I20250113 03:07:26.182075 24077 consensus_queue.cc:237] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:26.182567 23776 catalog_manager.cc:5526] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c483e6dd58240a499832e28c90ef370 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "4c483e6dd58240a499832e28c90ef370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:26.229810 23879 raft_consensus.cc:1270] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 1 FOLLOWER]: Refusing update from remote peer 4c483e6dd58240a499832e28c90ef370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:26.229812 23955 raft_consensus.cc:1270] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 1 FOLLOWER]: Refusing update from remote peer 4c483e6dd58240a499832e28c90ef370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:26.230087 24077 consensus_queue.cc:1035] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:26.230216 24081 consensus_queue.cc:1035] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:26.231611 24088 mvcc.cc:204] Tried to move back new op lower bound from 7113677398957314048 to 7113677398762205184. Current Snapshot: MvccSnapshot[applied={T|T < 7113677398955986944}]
I20250113 03:07:36.689194 24106 raft_consensus.cc:988] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: : Instructing follower 5a8134bb26634adc9cbe9846b0bde564 to start an election
I20250113 03:07:36.689270 24106 raft_consensus.cc:1076] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 LEADER]: Signalling peer 5a8134bb26634adc9cbe9846b0bde564 to start an election
I20250113 03:07:36.689517 23879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23"
dest_uuid: "5a8134bb26634adc9cbe9846b0bde564"
from {username='slave'} at 127.0.0.1:39264
I20250113 03:07:36.689590 23879 raft_consensus.cc:491] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:07:36.689615 23879 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:36.690270 23879 raft_consensus.cc:513] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:36.690441 23879 leader_election.cc:290] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [CANDIDATE]: Term 2 election: Requested vote from peers 4c483e6dd58240a499832e28c90ef370 (127.20.18.67:34307), 2b094dbd07a141a58373d6b1085aad61 (127.20.18.66:35965)
I20250113 03:07:36.692955 23955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "5a8134bb26634adc9cbe9846b0bde564" candidate_term: 2 candidate_status { last_received { term: 1 index: 15597 } } ignore_live_leader: true dest_uuid: "2b094dbd07a141a58373d6b1085aad61"
I20250113 03:07:36.693037 23955 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:36.692960 24030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aadaee7eab64b119a62c6b3e8d9ef23" candidate_uuid: "5a8134bb26634adc9cbe9846b0bde564" candidate_term: 2 candidate_status { last_received { term: 1 index: 15597 } } ignore_live_leader: true dest_uuid: "4c483e6dd58240a499832e28c90ef370"
I20250113 03:07:36.693080 24030 raft_consensus.cc:3049] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:07:36.693116 24030 raft_consensus.cc:738] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c483e6dd58240a499832e28c90ef370, State: Running, Role: LEADER
I20250113 03:07:36.693167 24030 consensus_queue.cc:260] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15597, Committed index: 15597, Last appended: 1.15597, Last appended by leader: 15597, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:36.693243 24030 raft_consensus.cc:3054] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:36.693660 23955 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8134bb26634adc9cbe9846b0bde564 in term 2.
I20250113 03:07:36.693861 23845 leader_election.cc:304] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b094dbd07a141a58373d6b1085aad61, 5a8134bb26634adc9cbe9846b0bde564; no voters:
I20250113 03:07:36.693858 24030 raft_consensus.cc:2463] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8134bb26634adc9cbe9846b0bde564 in term 2.
I20250113 03:07:36.694003 24126 raft_consensus.cc:2798] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:07:36.694142 24126 raft_consensus.cc:695] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 LEADER]: Becoming Leader. State: Replica: 5a8134bb26634adc9cbe9846b0bde564, State: Running, Role: LEADER
I20250113 03:07:36.694204 24126 consensus_queue.cc:237] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15597, Committed index: 15597, Last appended: 1.15597, Last appended by leader: 15597, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } }
I20250113 03:07:36.694617 23769 catalog_manager.cc:5526] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 reported cstate change: term changed from 1 to 2, leader changed from 4c483e6dd58240a499832e28c90ef370 (127.20.18.67) to 5a8134bb26634adc9cbe9846b0bde564 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "5a8134bb26634adc9cbe9846b0bde564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8134bb26634adc9cbe9846b0bde564" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c483e6dd58240a499832e28c90ef370" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b094dbd07a141a58373d6b1085aad61" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35965 } health_report { overall_health: UNKNOWN } } }
W20250113 03:07:36.701797 23945 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45612: Illegal state: replica 2b094dbd07a141a58373d6b1085aad61 is not leader of this config: current role FOLLOWER
W20250113 03:07:36.702142 23945 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45612: Illegal state: replica 2b094dbd07a141a58373d6b1085aad61 is not leader of this config: current role FOLLOWER
W20250113 03:07:36.702499 23945 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45612: Illegal state: replica 2b094dbd07a141a58373d6b1085aad61 is not leader of this config: current role FOLLOWER
I20250113 03:07:36.702708 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:07:36.705379 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:36.705461 20553 tablet_replica.cc:331] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: stopping tablet replica
I20250113 03:07:36.705513 20553 raft_consensus.cc:2238] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:07:36.705580 20553 pending_rounds.cc:62] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Trying to abort 1 pending ops.
I20250113 03:07:36.705636 20553 pending_rounds.cc:69] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564: Aborting op as it isn't in flight: id { term: 2 index: 15598 } timestamp: 7113677441819938816 op_type: NO_OP noop_request { }
I20250113 03:07:36.705701 20553 raft_consensus.cc:2883] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:07:36.705729 20553 raft_consensus.cc:2267] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 5a8134bb26634adc9cbe9846b0bde564 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:36.720000 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:07:36.720909 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:07:36.723706 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:36.723802 20553 tablet_replica.cc:331] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61: stopping tablet replica
I20250113 03:07:36.723852 20553 raft_consensus.cc:2238] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:36.723912 20553 raft_consensus.cc:2267] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 2b094dbd07a141a58373d6b1085aad61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:36.743520 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:07:36.744197 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:07:36.746601 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:36.746707 20553 tablet_replica.cc:331] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370: stopping tablet replica
I20250113 03:07:36.746748 20553 raft_consensus.cc:2238] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:36.746789 20553 raft_consensus.cc:2267] T 8aadaee7eab64b119a62c6b3e8d9ef23 P 4c483e6dd58240a499832e28c90ef370 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:36.758473 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:07:36.759150 20553 master.cc:537] Master@127.20.18.126:42883 shutting down...
I20250113 03:07:36.761584 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:36.761648 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:36.761667 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P b9f190c122114178aa03b0a20094b8ca: stopping tablet replica
I20250113 03:07:36.773099 20553 master.cc:559] Master@127.20.18.126:42883 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyNonLeader (11043 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletBootstrapping
I20250113 03:07:36.793114 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:44597
I20250113 03:07:36.793260 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:36.794494 24132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.794813 24135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:36.795449 20553 server_base.cc:1034] running on GCE node
W20250113 03:07:36.795574 24133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:36.795709 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:36.795742 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:36.795761 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737656795761 us; error 0 us; skew 500 ppm
I20250113 03:07:36.796348 20553 webserver.cc:458] Webserver started at http://127.20.18.126:43125/ using document root <none> and password file <none>
I20250113 03:07:36.796434 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:36.796481 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:36.796568 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:36.796840 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "53b9c3afd88945c49d0e50d6a9fb069a"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.797668 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:36.798050 24140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.798168 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.798207 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "53b9c3afd88945c49d0e50d6a9fb069a"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.798252 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:36.819842 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:36.820024 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:36.822412 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:44597
I20250113 03:07:36.823770 24192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:44597 every 8 connection(s)
I20250113 03:07:36.824664 24193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:36.825696 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a: Bootstrap starting.
I20250113 03:07:36.825987 24193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:36.826431 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a: No bootstrap required, opened a new log
I20250113 03:07:36.826541 24193 raft_consensus.cc:357] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER }
I20250113 03:07:36.826587 24193 raft_consensus.cc:383] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:36.826609 24193 raft_consensus.cc:738] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b9c3afd88945c49d0e50d6a9fb069a, State: Initialized, Role: FOLLOWER
I20250113 03:07:36.826666 24193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER }
I20250113 03:07:36.826699 24193 raft_consensus.cc:397] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:07:36.826720 24193 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:07:36.826747 24193 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:36.827171 24193 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER }
I20250113 03:07:36.827224 24193 leader_election.cc:304] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53b9c3afd88945c49d0e50d6a9fb069a; no voters:
I20250113 03:07:36.827316 24193 leader_election.cc:290] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:07:36.827369 24196 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:36.827462 24193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:07:36.827502 24196 raft_consensus.cc:695] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 1 LEADER]: Becoming Leader. State: Replica: 53b9c3afd88945c49d0e50d6a9fb069a, State: Running, Role: LEADER
I20250113 03:07:36.827557 24196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER }
I20250113 03:07:36.827720 24196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53b9c3afd88945c49d0e50d6a9fb069a. Latest consensus state: current_term: 1 leader_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER } }
I20250113 03:07:36.827745 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b9c3afd88945c49d0e50d6a9fb069a" member_type: VOTER } }
I20250113 03:07:36.827847 24196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:36.827880 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:36.828074 24203 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:07:36.828239 24203 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:07:36.828783 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:07:36.828790 24203 catalog_manager.cc:1348] Generated new cluster ID: 7751c07fd6b14196bc64ec0af1512926
I20250113 03:07:36.828884 24203 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:07:36.844640 24203 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:07:36.845006 24203 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:07:36.848134 24203 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a: Generated new TSK 0
I20250113 03:07:36.848202 24203 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:07:36.860471 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:36.861843 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.862066 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.862736 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:36.863303 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:36.863374 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:36.863420 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:36.863438 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737656863439 us; error 0 us; skew 500 ppm
I20250113 03:07:36.864056 20553 webserver.cc:458] Webserver started at http://127.20.18.65:37901/ using document root <none> and password file <none>
I20250113 03:07:36.864123 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:36.864171 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:36.864224 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:36.864478 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "1e4ee0b8e8c04e2e81821fb529c4de10"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.865303 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:36.865727 24222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.865859 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.865904 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "1e4ee0b8e8c04e2e81821fb529c4de10"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.865937 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:36.878664 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:36.878798 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:36.878911 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:36.879050 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:36.879071 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.879093 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:36.879109 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.881681 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:40669
I20250113 03:07:36.881707 24285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:40669 every 8 connection(s)
I20250113 03:07:36.882000 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:36.883095 24290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.883291 24291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.883875 24293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:36.884488 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:36.884611 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:36.884654 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:36.884668 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737656884668 us; error 0 us; skew 500 ppm
I20250113 03:07:36.885449 24286 heartbeater.cc:346] Connected to a master server at 127.20.18.126:44597
I20250113 03:07:36.885485 20553 webserver.cc:458] Webserver started at http://127.20.18.66:38361/ using document root <none> and password file <none>
I20250113 03:07:36.885514 24286 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:36.885565 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:36.885608 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:36.885618 24286 heartbeater.cc:510] Master 127.20.18.126:44597 requested a full tablet report, sending...
I20250113 03:07:36.885663 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:36.885848 24157 ts_manager.cc:194] Registered new tserver with Master: 1e4ee0b8e8c04e2e81821fb529c4de10 (127.20.18.65:40669)
I20250113 03:07:36.886032 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "21d105bd01144a02b3dfe8a88ec14882"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.886652 24157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60800
I20250113 03:07:36.887063 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:36.887459 24298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.887575 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.887615 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "21d105bd01144a02b3dfe8a88ec14882"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.887668 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:36.905613 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:36.905774 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:36.905903 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:36.906077 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:36.906101 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.906131 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:36.906155 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.908692 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:44517
I20250113 03:07:36.908731 24361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:44517 every 8 connection(s)
I20250113 03:07:36.909021 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:36.910213 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.910440 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:36.911051 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:36.911720 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:36.911821 24362 heartbeater.cc:346] Connected to a master server at 127.20.18.126:44597
I20250113 03:07:36.911851 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:36.911873 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:36.911875 24362 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:36.911908 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737656911908 us; error 0 us; skew 500 ppm
I20250113 03:07:36.912053 24362 heartbeater.cc:510] Master 127.20.18.126:44597 requested a full tablet report, sending...
I20250113 03:07:36.912281 24157 ts_manager.cc:194] Registered new tserver with Master: 21d105bd01144a02b3dfe8a88ec14882 (127.20.18.66:44517)
I20250113 03:07:36.912600 20553 webserver.cc:458] Webserver started at http://127.20.18.67:45255/ using document root <none> and password file <none>
I20250113 03:07:36.912685 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:36.912703 24157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60812
I20250113 03:07:36.912725 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:36.912808 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:36.913084 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "c780529ae18c4b88bb02bb3846be4c69"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.913909 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:36.914284 24373 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.914407 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.914438 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "c780529ae18c4b88bb02bb3846be4c69"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:36.914470 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:36.924742 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:36.924875 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:36.924978 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:36.925103 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:36.925123 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.925148 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:36.925163 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.927932 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:35565
I20250113 03:07:36.927978 24436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:35565 every 8 connection(s)
I20250113 03:07:36.930625 24437 heartbeater.cc:346] Connected to a master server at 127.20.18.126:44597
I20250113 03:07:36.930678 24437 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:36.930771 24437 heartbeater.cc:510] Master 127.20.18.126:44597 requested a full tablet report, sending...
I20250113 03:07:36.930972 24157 ts_manager.cc:194] Registered new tserver with Master: c780529ae18c4b88bb02bb3846be4c69 (127.20.18.67:35565)
I20250113 03:07:36.931265 24157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60822
I20250113 03:07:36.931368 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003296745s
I20250113 03:07:36.931667 20553 test_util.cc:274] Using random seed: -1088682669
I20250113 03:07:36.935448 24157 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60824:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:07:36.935555 24157 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:07:36.939603 24401 tablet_service.cc:1467] Processing CreateTablet for tablet 3e253bd667fb4f2a856e4858b0e79b55 (DEFAULT_TABLE table=test-workload [id=6da7a5d1ecf6413da910f96aedeaa38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:36.939730 24401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e253bd667fb4f2a856e4858b0e79b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:36.940120 24250 tablet_service.cc:1467] Processing CreateTablet for tablet 3e253bd667fb4f2a856e4858b0e79b55 (DEFAULT_TABLE table=test-workload [id=6da7a5d1ecf6413da910f96aedeaa38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:36.940209 24250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e253bd667fb4f2a856e4858b0e79b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:36.940332 24326 tablet_service.cc:1467] Processing CreateTablet for tablet 3e253bd667fb4f2a856e4858b0e79b55 (DEFAULT_TABLE table=test-workload [id=6da7a5d1ecf6413da910f96aedeaa38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:36.940409 24326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e253bd667fb4f2a856e4858b0e79b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:36.941521 24453 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap starting.
I20250113 03:07:36.941695 24454 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: Bootstrap starting.
I20250113 03:07:36.941752 24452 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: Bootstrap starting.
I20250113 03:07:36.941980 24453 tablet_bootstrap.cc:654] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:36.942122 24454 tablet_bootstrap.cc:654] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:36.942193 24452 tablet_bootstrap.cc:654] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:36.942561 24453 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: No bootstrap required, opened a new log
I20250113 03:07:36.942605 24453 ts_tablet_manager.cc:1397] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:36.942700 24454 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: No bootstrap required, opened a new log
I20250113 03:07:36.942708 24452 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: No bootstrap required, opened a new log
I20250113 03:07:36.942759 24452 ts_tablet_manager.cc:1397] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:36.942754 24454 ts_tablet_manager.cc:1397] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:36.942747 24453 raft_consensus.cc:357] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.942803 24453 raft_consensus.cc:383] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:36.942817 24453 raft_consensus.cc:738] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4ee0b8e8c04e2e81821fb529c4de10, State: Initialized, Role: FOLLOWER
I20250113 03:07:36.942857 24453 consensus_queue.cc:260] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.942917 24452 raft_consensus.cc:357] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.942917 24454 raft_consensus.cc:357] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.943007 24452 raft_consensus.cc:383] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:36.943022 24454 raft_consensus.cc:383] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:36.942993 24453 ts_tablet_manager.cc:1428] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.943055 24454 raft_consensus.cc:738] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d105bd01144a02b3dfe8a88ec14882, State: Initialized, Role: FOLLOWER
I20250113 03:07:36.943120 24454 consensus_queue.cc:260] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.943241 24454 ts_tablet_manager.cc:1428] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:36.943290 24362 heartbeater.cc:502] Master 127.20.18.126:44597 was elected leader, sending a full tablet report...
I20250113 03:07:36.943030 24452 raft_consensus.cc:738] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c780529ae18c4b88bb02bb3846be4c69, State: Initialized, Role: FOLLOWER
I20250113 03:07:36.943295 24286 heartbeater.cc:502] Master 127.20.18.126:44597 was elected leader, sending a full tablet report...
I20250113 03:07:36.943377 24452 consensus_queue.cc:260] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.943531 24452 ts_tablet_manager.cc:1428] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:36.943634 24437 heartbeater.cc:502] Master 127.20.18.126:44597 was elected leader, sending a full tablet report...
I20250113 03:07:36.988497 24459 raft_consensus.cc:491] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:07:36.988619 24459 raft_consensus.cc:513] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.988806 24459 leader_election.cc:290] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4ee0b8e8c04e2e81821fb529c4de10 (127.20.18.65:40669), c780529ae18c4b88bb02bb3846be4c69 (127.20.18.67:35565)
I20250113 03:07:36.991273 24260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "21d105bd01144a02b3dfe8a88ec14882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" is_pre_election: true
I20250113 03:07:36.991348 24260 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d105bd01144a02b3dfe8a88ec14882 in term 0.
I20250113 03:07:36.991379 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "21d105bd01144a02b3dfe8a88ec14882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c780529ae18c4b88bb02bb3846be4c69" is_pre_election: true
I20250113 03:07:36.991468 24411 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d105bd01144a02b3dfe8a88ec14882 in term 0.
I20250113 03:07:36.991508 24299 leader_election.cc:304] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4ee0b8e8c04e2e81821fb529c4de10, 21d105bd01144a02b3dfe8a88ec14882; no voters:
I20250113 03:07:36.991621 24459 raft_consensus.cc:2798] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:07:36.991657 24459 raft_consensus.cc:491] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:07:36.991670 24459 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:36.992244 24459 raft_consensus.cc:513] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.992357 24459 leader_election.cc:290] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4ee0b8e8c04e2e81821fb529c4de10 (127.20.18.65:40669), c780529ae18c4b88bb02bb3846be4c69 (127.20.18.67:35565)
I20250113 03:07:36.992480 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "21d105bd01144a02b3dfe8a88ec14882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c780529ae18c4b88bb02bb3846be4c69"
I20250113 03:07:36.992556 24411 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:36.992496 24260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "21d105bd01144a02b3dfe8a88ec14882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10"
I20250113 03:07:36.992588 24260 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:36.993161 24411 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d105bd01144a02b3dfe8a88ec14882 in term 1.
I20250113 03:07:36.993197 24260 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d105bd01144a02b3dfe8a88ec14882 in term 1.
I20250113 03:07:36.993320 24302 leader_election.cc:304] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d105bd01144a02b3dfe8a88ec14882, c780529ae18c4b88bb02bb3846be4c69; no voters:
I20250113 03:07:36.993419 24459 raft_consensus.cc:2798] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:36.993570 24459 raft_consensus.cc:695] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 LEADER]: Becoming Leader. State: Replica: 21d105bd01144a02b3dfe8a88ec14882, State: Running, Role: LEADER
I20250113 03:07:36.993638 24459 consensus_queue.cc:237] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:36.994102 24156 catalog_manager.cc:5526] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d105bd01144a02b3dfe8a88ec14882 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "21d105bd01144a02b3dfe8a88ec14882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } health_report { overall_health: HEALTHY } } }
I20250113 03:07:37.004987 24260 raft_consensus.cc:1270] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 1 FOLLOWER]: Refusing update from remote peer 21d105bd01144a02b3dfe8a88ec14882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:37.005016 24411 raft_consensus.cc:1270] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 1 FOLLOWER]: Refusing update from remote peer 21d105bd01144a02b3dfe8a88ec14882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:37.005158 24462 consensus_queue.cc:1035] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:37.005264 24459 consensus_queue.cc:1035] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:47.504948 24500 raft_consensus.cc:988] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: : Instructing follower 1e4ee0b8e8c04e2e81821fb529c4de10 to start an election
I20250113 03:07:47.505019 24500 raft_consensus.cc:1076] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 LEADER]: Signalling peer 1e4ee0b8e8c04e2e81821fb529c4de10 to start an election
I20250113 03:07:47.505153 24259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55"
dest_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10"
from {username='slave'} at 127.0.0.1:48078
I20250113 03:07:47.505227 24259 raft_consensus.cc:491] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:07:47.505254 24259 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:47.505817 24259 raft_consensus.cc:513] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:47.505965 24259 leader_election.cc:290] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [CANDIDATE]: Term 2 election: Requested vote from peers c780529ae18c4b88bb02bb3846be4c69 (127.20.18.67:35565), 21d105bd01144a02b3dfe8a88ec14882 (127.20.18.66:44517)
I20250113 03:07:47.508989 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" candidate_term: 2 candidate_status { last_received { term: 1 index: 16002 } } ignore_live_leader: true dest_uuid: "c780529ae18c4b88bb02bb3846be4c69"
I20250113 03:07:47.509054 24410 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:47.508998 24336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e253bd667fb4f2a856e4858b0e79b55" candidate_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" candidate_term: 2 candidate_status { last_received { term: 1 index: 16002 } } ignore_live_leader: true dest_uuid: "21d105bd01144a02b3dfe8a88ec14882"
I20250113 03:07:47.509097 24336 raft_consensus.cc:3049] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:07:47.509130 24336 raft_consensus.cc:738] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21d105bd01144a02b3dfe8a88ec14882, State: Running, Role: LEADER
I20250113 03:07:47.509186 24336 consensus_queue.cc:260] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16002, Committed index: 16002, Last appended: 1.16002, Last appended by leader: 16002, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:47.509258 24336 raft_consensus.cc:3054] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:07:47.509588 24410 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4ee0b8e8c04e2e81821fb529c4de10 in term 2.
I20250113 03:07:47.509757 24226 leader_election.cc:304] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4ee0b8e8c04e2e81821fb529c4de10, c780529ae18c4b88bb02bb3846be4c69; no voters:
I20250113 03:07:47.509814 24336 raft_consensus.cc:2463] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4ee0b8e8c04e2e81821fb529c4de10 in term 2.
I20250113 03:07:47.509936 24510 raft_consensus.cc:2798] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:07:47.510062 24510 raft_consensus.cc:695] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 LEADER]: Becoming Leader. State: Replica: 1e4ee0b8e8c04e2e81821fb529c4de10, State: Running, Role: LEADER
I20250113 03:07:47.510115 24510 consensus_queue.cc:237] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16002, Committed index: 16002, Last appended: 1.16002, Last appended by leader: 16002, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:47.510490 24156 catalog_manager.cc:5526] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 reported cstate change: term changed from 1 to 2, leader changed from 21d105bd01144a02b3dfe8a88ec14882 (127.20.18.66) to 1e4ee0b8e8c04e2e81821fb529c4de10 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:47.516595 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:07:47.519047 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:47.519133 20553 tablet_replica.cc:331] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: stopping tablet replica
I20250113 03:07:47.519189 20553 raft_consensus.cc:2238] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:07:47.519248 20553 pending_rounds.cc:62] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Trying to abort 1 pending ops.
I20250113 03:07:47.519269 20553 pending_rounds.cc:69] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Aborting op as it isn't in flight: id { term: 2 index: 16003 } timestamp: 7113677486121873408 op_type: NO_OP noop_request { }
I20250113 03:07:47.519311 20553 raft_consensus.cc:2883] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:07:47.519338 20553 raft_consensus.cc:2267] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:47.536677 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:07:47.537540 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:47.539103 24519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:47.539273 24516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:47.539824 24517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:47.540431 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:47.540547 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:47.540577 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:47.540597 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737667540597 us; error 0 us; skew 500 ppm
I20250113 03:07:47.541215 20553 webserver.cc:458] Webserver started at http://127.20.18.65:37901/ using document root <none> and password file <none>
I20250113 03:07:47.541311 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:47.541344 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:07:47.541844 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250113 03:07:47.542241 24524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:47.542348 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:47.542385 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "1e4ee0b8e8c04e2e81821fb529c4de10"
format_stamp: "Formatted at 2025-01-13 03:07:36 on dist-test-slave-d4dl"
I20250113 03:07:47.542433 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:47.552742 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:47.552879 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:47.552992 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:47.553253 24531 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:07:47.553866 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:07:47.553896 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:07:47.553934 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:07:47.554256 20553 ts_tablet_manager.cc:610] Registered 1 tablets
W20250113 03:07:47.554277 24531 ts_tablet_manager.cc:1376] Injecting 1000ms delay in tablet bootstrapping
I20250113 03:07:47.554287 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:47.556600 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:40669
I20250113 03:07:47.557562 24588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:40669 every 8 connection(s)
I20250113 03:07:47.561579 24589 heartbeater.cc:346] Connected to a master server at 127.20.18.126:44597
I20250113 03:07:47.561676 24589 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:47.561779 24589 heartbeater.cc:510] Master 127.20.18.126:44597 requested a full tablet report, sending...
I20250113 03:07:47.561997 24156 ts_manager.cc:194] Re-registered known tserver with Master: 1e4ee0b8e8c04e2e81821fb529c4de10 (127.20.18.65:40669)
I20250113 03:07:47.562150 20553 tablet_server.cc:178] TabletServer@127.20.18.65:40669 shutting down...
I20250113 03:07:47.562389 24156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52172
I20250113 03:07:47.564010 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:48.554616 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap starting.
W20250113 03:07:49.067489 24596 raft_consensus.cc:668] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: failed to trigger leader election: Illegal state: leader elections are disabled
I20250113 03:07:49.189186 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap replayed 1/4 log segments. Stats: ops{read=4943 overwritten=0 applied=4941 ignored=0} inserts{seen=247000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:07:49.897604 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap replayed 2/4 log segments. Stats: ops{read=9888 overwritten=0 applied=9884 ignored=0} inserts{seen=494150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250113 03:07:50.648978 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap replayed 3/4 log segments. Stats: ops{read=14827 overwritten=0 applied=14824 ignored=0} inserts{seen=741150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:07:50.829322 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap replayed 4/4 log segments. Stats: ops{read=16003 overwritten=0 applied=16002 ignored=0} inserts{seen=800050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:07:50.829794 24531 tablet_bootstrap.cc:492] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Bootstrap complete.
I20250113 03:07:50.834370 24531 ts_tablet_manager.cc:1397] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Time spent bootstrapping tablet: real 2.280s user 1.946s sys 0.332s
I20250113 03:07:50.834537 24531 raft_consensus.cc:357] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:50.834609 24531 raft_consensus.cc:738] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4ee0b8e8c04e2e81821fb529c4de10, State: Initialized, Role: FOLLOWER
I20250113 03:07:50.834657 24531 consensus_queue.cc:260] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16002, Last appended: 2.16003, Last appended by leader: 16003, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4ee0b8e8c04e2e81821fb529c4de10" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 40669 } } peers { permanent_uuid: "c780529ae18c4b88bb02bb3846be4c69" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 35565 } } peers { permanent_uuid: "21d105bd01144a02b3dfe8a88ec14882" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 44517 } }
I20250113 03:07:50.834813 24531 ts_tablet_manager.cc:1428] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250113 03:07:50.834944 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:07:50.835021 20553 raft_consensus.cc:2238] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:50.835060 20553 pending_rounds.cc:62] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Trying to abort 1 pending ops.
I20250113 03:07:50.835083 20553 pending_rounds.cc:69] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10: Aborting op as it isn't in flight: id { term: 2 index: 16003 } timestamp: 7113677486121873408 op_type: NO_OP noop_request { }
I20250113 03:07:50.835126 20553 raft_consensus.cc:2883] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:07:50.835155 20553 raft_consensus.cc:2267] T 3e253bd667fb4f2a856e4858b0e79b55 P 1e4ee0b8e8c04e2e81821fb529c4de10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:50.847997 20553 tablet_server.cc:195] TabletServer@127.20.18.65:40669 shutdown complete.
I20250113 03:07:50.849105 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:07:50.851817 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:50.851920 20553 tablet_replica.cc:331] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882: stopping tablet replica
I20250113 03:07:50.851986 20553 raft_consensus.cc:2238] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:50.852027 20553 raft_consensus.cc:2267] T 3e253bd667fb4f2a856e4858b0e79b55 P 21d105bd01144a02b3dfe8a88ec14882 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:50.871094 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:07:50.871919 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:07:50.874756 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:07:50.874856 20553 tablet_replica.cc:331] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69: stopping tablet replica
I20250113 03:07:50.874907 20553 raft_consensus.cc:2238] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:07:50.874965 20553 raft_consensus.cc:2267] T 3e253bd667fb4f2a856e4858b0e79b55 P c780529ae18c4b88bb02bb3846be4c69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:50.896811 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:07:50.897667 20553 master.cc:537] Master@127.20.18.126:44597 shutting down...
I20250113 03:07:50.900334 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:07:50.900400 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:07:50.900418 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 53b9c3afd88945c49d0e50d6a9fb069a: stopping tablet replica
I20250113 03:07:50.912038 20553 master.cc:559] Master@127.20.18.126:44597 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletBootstrapping (14138 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotRunning
I20250113 03:07:50.931447 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:36903
I20250113 03:07:50.931607 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:07:50.933331 20553 server_base.cc:1034] running on GCE node
W20250113 03:07:50.933983 24607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:50.934487 24605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:50.934792 24604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:50.934935 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:50.934973 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:50.934985 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737670934986 us; error 0 us; skew 500 ppm
I20250113 03:07:50.939380 20553 webserver.cc:458] Webserver started at http://127.20.18.126:38301/ using document root <none> and password file <none>
I20250113 03:07:50.939476 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:50.939533 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:50.939579 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:50.939843 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "ba973badfeed4c9cb6f3c80e0c304287"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:50.940702 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:50.941138 24612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.941263 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.941303 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "ba973badfeed4c9cb6f3c80e0c304287"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:50.941340 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:50.957329 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:50.957506 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:50.959784 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:36903
I20250113 03:07:50.962584 24664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:36903 every 8 connection(s)
I20250113 03:07:50.962692 24665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:50.963600 24665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287: Bootstrap starting.
I20250113 03:07:50.963850 24665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:50.964246 24665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287: No bootstrap required, opened a new log
I20250113 03:07:50.964342 24665 raft_consensus.cc:357] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER }
I20250113 03:07:50.964402 24665 raft_consensus.cc:383] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:50.964416 24665 raft_consensus.cc:738] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba973badfeed4c9cb6f3c80e0c304287, State: Initialized, Role: FOLLOWER
I20250113 03:07:50.964447 24665 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER }
I20250113 03:07:50.964475 24665 raft_consensus.cc:397] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:07:50.964495 24665 raft_consensus.cc:491] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:07:50.964517 24665 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:50.965008 24665 raft_consensus.cc:513] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER }
I20250113 03:07:50.965059 24665 leader_election.cc:304] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba973badfeed4c9cb6f3c80e0c304287; no voters:
I20250113 03:07:50.965152 24665 leader_election.cc:290] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:07:50.965195 24668 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:50.965282 24665 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:07:50.965348 24668 raft_consensus.cc:695] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 1 LEADER]: Becoming Leader. State: Replica: ba973badfeed4c9cb6f3c80e0c304287, State: Running, Role: LEADER
I20250113 03:07:50.965404 24668 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER }
I20250113 03:07:50.965621 24669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba973badfeed4c9cb6f3c80e0c304287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER } }
I20250113 03:07:50.965646 24670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba973badfeed4c9cb6f3c80e0c304287. Latest consensus state: current_term: 1 leader_uuid: "ba973badfeed4c9cb6f3c80e0c304287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba973badfeed4c9cb6f3c80e0c304287" member_type: VOTER } }
I20250113 03:07:50.965695 24669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:50.965705 24670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [sys.catalog]: This master's current role is: LEADER
I20250113 03:07:50.965948 24675 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:07:50.966073 24675 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:07:50.966490 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:07:50.966526 24675 catalog_manager.cc:1348] Generated new cluster ID: 2130c47a01b8414daeb39e9f8f79644c
I20250113 03:07:50.966553 24675 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:07:50.970638 24675 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:07:50.971011 24675 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:07:50.976433 24675 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287: Generated new TSK 0
I20250113 03:07:50.976493 24675 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:07:50.982136 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:50.983461 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:50.984094 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:50.984699 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:50.985211 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:50.985327 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:50.985361 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:50.985380 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737670985380 us; error 0 us; skew 500 ppm
I20250113 03:07:50.985980 20553 webserver.cc:458] Webserver started at http://127.20.18.65:38287/ using document root <none> and password file <none>
I20250113 03:07:50.986059 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:50.986104 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:50.986155 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:50.986420 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "118deb59b33541528dced43a0ac62bf4"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:50.987200 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:50.987586 24694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.987699 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.987735 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "118deb59b33541528dced43a0ac62bf4"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:50.987779 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:50.992432 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:50.992600 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:50.992714 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:50.992887 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:50.992910 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.992940 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:50.992964 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:50.995344 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:38401
I20250113 03:07:50.995405 24757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:38401 every 8 connection(s)
I20250113 03:07:50.995662 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:50.997465 24762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:50.997879 24763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:50.998492 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:50.998749 24758 heartbeater.cc:346] Connected to a master server at 127.20.18.126:36903
I20250113 03:07:50.998795 24758 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:50.998874 24758 heartbeater.cc:510] Master 127.20.18.126:36903 requested a full tablet report, sending...
W20250113 03:07:50.998997 24765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:50.999101 24629 ts_manager.cc:194] Registered new tserver with Master: 118deb59b33541528dced43a0ac62bf4 (127.20.18.65:38401)
I20250113 03:07:50.999130 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:50.999159 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:50.999171 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737670999172 us; error 0 us; skew 500 ppm
I20250113 03:07:50.999810 24629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60870
I20250113 03:07:50.999856 20553 webserver.cc:458] Webserver started at http://127.20.18.66:46401/ using document root <none> and password file <none>
I20250113 03:07:50.999912 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:50.999941 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:51.000012 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:51.000247 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "7b7e45ba3ecc4721a0a5be32e15892ce"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:51.001067 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:07:51.001444 24770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.001560 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.001601 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "7b7e45ba3ecc4721a0a5be32e15892ce"
format_stamp: "Formatted at 2025-01-13 03:07:50 on dist-test-slave-d4dl"
I20250113 03:07:51.001638 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:51.013648 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:51.013782 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:51.013885 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:51.014010 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:51.014031 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.014053 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:51.014070 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.016470 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:35213
I20250113 03:07:51.016549 24834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:35213 every 8 connection(s)
I20250113 03:07:51.016795 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:07:51.017933 24838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:51.018533 24841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:07:51.019021 24839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:07:51.019440 24835 heartbeater.cc:346] Connected to a master server at 127.20.18.126:36903
I20250113 03:07:51.019495 24835 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:51.019577 20553 server_base.cc:1034] running on GCE node
I20250113 03:07:51.019614 24835 heartbeater.cc:510] Master 127.20.18.126:36903 requested a full tablet report, sending...
I20250113 03:07:51.019683 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:07:51.019722 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:07:51.019740 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737671019740 us; error 0 us; skew 500 ppm
I20250113 03:07:51.019824 24629 ts_manager.cc:194] Registered new tserver with Master: 7b7e45ba3ecc4721a0a5be32e15892ce (127.20.18.66:35213)
I20250113 03:07:51.020291 20553 webserver.cc:458] Webserver started at http://127.20.18.67:36537/ using document root <none> and password file <none>
I20250113 03:07:51.020362 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:07:51.020396 24629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60872
I20250113 03:07:51.020404 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:07:51.020489 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:07:51.020756 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "f00ce5a72049484a81629b9bc4bb358c"
format_stamp: "Formatted at 2025-01-13 03:07:51 on dist-test-slave-d4dl"
I20250113 03:07:51.021550 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:51.021937 24846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.022060 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.022099 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "f00ce5a72049484a81629b9bc4bb358c"
format_stamp: "Formatted at 2025-01-13 03:07:51 on dist-test-slave-d4dl"
I20250113 03:07:51.022141 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:07:51.041193 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:07:51.041364 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:07:51.041504 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:07:51.041678 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:07:51.041702 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.041731 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:07:51.041755 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.044556 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:44365
I20250113 03:07:51.044595 24909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:44365 every 8 connection(s)
I20250113 03:07:51.047055 24910 heartbeater.cc:346] Connected to a master server at 127.20.18.126:36903
I20250113 03:07:51.047101 24910 heartbeater.cc:463] Registering TS with master...
I20250113 03:07:51.047185 24910 heartbeater.cc:510] Master 127.20.18.126:36903 requested a full tablet report, sending...
I20250113 03:07:51.047385 24629 ts_manager.cc:194] Registered new tserver with Master: f00ce5a72049484a81629b9bc4bb358c (127.20.18.67:44365)
I20250113 03:07:51.047683 24629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60888
I20250113 03:07:51.048020 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003294482s
I20250113 03:07:51.048309 20553 test_util.cc:274] Using random seed: -1074566027
I20250113 03:07:51.052286 24629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60892:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:07:51.052459 24629 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:07:51.056991 24874 tablet_service.cc:1467] Processing CreateTablet for tablet 5f936600153e46e59929870a2ec0e050 (DEFAULT_TABLE table=test-workload [id=82a9bdf803224a52aad7efd101a0d59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:51.057116 24874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f936600153e46e59929870a2ec0e050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:51.057384 24799 tablet_service.cc:1467] Processing CreateTablet for tablet 5f936600153e46e59929870a2ec0e050 (DEFAULT_TABLE table=test-workload [id=82a9bdf803224a52aad7efd101a0d59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:51.057469 24799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f936600153e46e59929870a2ec0e050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:51.057634 24722 tablet_service.cc:1467] Processing CreateTablet for tablet 5f936600153e46e59929870a2ec0e050 (DEFAULT_TABLE table=test-workload [id=82a9bdf803224a52aad7efd101a0d59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:07:51.057721 24722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f936600153e46e59929870a2ec0e050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:07:51.058591 24925 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: Bootstrap starting.
I20250113 03:07:51.058694 24926 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Bootstrap starting.
I20250113 03:07:51.058995 24925 tablet_bootstrap.cc:654] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:51.059091 24926 tablet_bootstrap.cc:654] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:51.059444 24927 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: Bootstrap starting.
I20250113 03:07:51.059604 24925 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: No bootstrap required, opened a new log
I20250113 03:07:51.059643 24925 ts_tablet_manager.cc:1397] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:51.059769 24925 raft_consensus.cc:357] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.059795 24927 tablet_bootstrap.cc:654] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: Neither blocks nor log segments found. Creating new log.
I20250113 03:07:51.059824 24925 raft_consensus.cc:383] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:51.059923 24925 raft_consensus.cc:738] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00ce5a72049484a81629b9bc4bb358c, State: Initialized, Role: FOLLOWER
I20250113 03:07:51.059980 24925 consensus_queue.cc:260] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.060091 24926 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: No bootstrap required, opened a new log
I20250113 03:07:51.060135 24926 ts_tablet_manager.cc:1397] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:51.060185 24925 ts_tablet_manager.cc:1428] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:51.060213 24910 heartbeater.cc:502] Master 127.20.18.126:36903 was elected leader, sending a full tablet report...
I20250113 03:07:51.060256 24926 raft_consensus.cc:357] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.060302 24926 raft_consensus.cc:383] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:51.060314 24926 raft_consensus.cc:738] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118deb59b33541528dced43a0ac62bf4, State: Initialized, Role: FOLLOWER
I20250113 03:07:51.060349 24926 consensus_queue.cc:260] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.060487 24926 ts_tablet_manager.cc:1428] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.060513 24927 tablet_bootstrap.cc:492] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: No bootstrap required, opened a new log
I20250113 03:07:51.060495 24758 heartbeater.cc:502] Master 127.20.18.126:36903 was elected leader, sending a full tablet report...
I20250113 03:07:51.060588 24927 ts_tablet_manager.cc:1397] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:07:51.060766 24927 raft_consensus.cc:357] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.060834 24927 raft_consensus.cc:383] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:07:51.060858 24927 raft_consensus.cc:738] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7e45ba3ecc4721a0a5be32e15892ce, State: Initialized, Role: FOLLOWER
I20250113 03:07:51.060921 24927 consensus_queue.cc:260] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.061067 24927 ts_tablet_manager.cc:1428] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:07:51.061137 24835 heartbeater.cc:502] Master 127.20.18.126:36903 was elected leader, sending a full tablet report...
I20250113 03:07:51.219430 24930 raft_consensus.cc:491] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:07:51.219549 24930 raft_consensus.cc:513] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.219785 24930 leader_election.cc:290] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7e45ba3ecc4721a0a5be32e15892ce (127.20.18.66:35213), 118deb59b33541528dced43a0ac62bf4 (127.20.18.65:38401)
I20250113 03:07:51.222791 24809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "f00ce5a72049484a81629b9bc4bb358c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" is_pre_election: true
I20250113 03:07:51.222805 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "f00ce5a72049484a81629b9bc4bb358c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "118deb59b33541528dced43a0ac62bf4" is_pre_election: true
I20250113 03:07:51.222908 24809 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00ce5a72049484a81629b9bc4bb358c in term 0.
I20250113 03:07:51.222923 24732 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f00ce5a72049484a81629b9bc4bb358c in term 0.
I20250113 03:07:51.223073 24848 leader_election.cc:304] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 118deb59b33541528dced43a0ac62bf4, f00ce5a72049484a81629b9bc4bb358c; no voters:
I20250113 03:07:51.223176 24930 raft_consensus.cc:2798] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:07:51.223229 24930 raft_consensus.cc:491] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:07:51.223251 24930 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:51.223840 24930 raft_consensus.cc:513] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.223963 24930 leader_election.cc:290] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [CANDIDATE]: Term 1 election: Requested vote from peers 7b7e45ba3ecc4721a0a5be32e15892ce (127.20.18.66:35213), 118deb59b33541528dced43a0ac62bf4 (127.20.18.65:38401)
I20250113 03:07:51.224081 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "f00ce5a72049484a81629b9bc4bb358c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "118deb59b33541528dced43a0ac62bf4"
I20250113 03:07:51.224134 24732 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:51.224593 24809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "f00ce5a72049484a81629b9bc4bb358c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce"
I20250113 03:07:51.224645 24809 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:07:51.224723 24732 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00ce5a72049484a81629b9bc4bb358c in term 1.
I20250113 03:07:51.224853 24848 leader_election.cc:304] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 118deb59b33541528dced43a0ac62bf4, f00ce5a72049484a81629b9bc4bb358c; no voters:
I20250113 03:07:51.224921 24930 raft_consensus.cc:2798] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:07:51.225054 24930 raft_consensus.cc:695] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 LEADER]: Becoming Leader. State: Replica: f00ce5a72049484a81629b9bc4bb358c, State: Running, Role: LEADER
I20250113 03:07:51.225109 24930 consensus_queue.cc:237] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:07:51.225224 24809 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f00ce5a72049484a81629b9bc4bb358c in term 1.
I20250113 03:07:51.225626 24628 catalog_manager.cc:5526] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c reported cstate change: term changed from 0 to 1, leader changed from <none> to f00ce5a72049484a81629b9bc4bb358c (127.20.18.67). New cstate: current_term: 1 leader_uuid: "f00ce5a72049484a81629b9bc4bb358c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } health_report { overall_health: UNKNOWN } } }
I20250113 03:07:51.239001 24732 raft_consensus.cc:1270] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 1 FOLLOWER]: Refusing update from remote peer f00ce5a72049484a81629b9bc4bb358c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:51.239035 24809 raft_consensus.cc:1270] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 1 FOLLOWER]: Refusing update from remote peer f00ce5a72049484a81629b9bc4bb358c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:07:51.239230 24935 consensus_queue.cc:1035] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [LEADER]: Connected to new peer: Peer: permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:07:51.239359 24930 consensus_queue.cc:1035] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:01.657258 24971 raft_consensus.cc:988] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: : Instructing follower 118deb59b33541528dced43a0ac62bf4 to start an election
I20250113 03:08:01.657330 24971 raft_consensus.cc:1076] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 LEADER]: Signalling peer 118deb59b33541528dced43a0ac62bf4 to start an election
I20250113 03:08:01.657521 24732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f936600153e46e59929870a2ec0e050"
dest_uuid: "118deb59b33541528dced43a0ac62bf4"
from {username='slave'} at 127.0.0.1:45364
I20250113 03:08:01.657596 24732 raft_consensus.cc:491] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:08:01.657614 24732 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:01.658180 24732 raft_consensus.cc:513] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:08:01.658326 24732 leader_election.cc:290] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [CANDIDATE]: Term 2 election: Requested vote from peers 7b7e45ba3ecc4721a0a5be32e15892ce (127.20.18.66:35213), f00ce5a72049484a81629b9bc4bb358c (127.20.18.67:44365)
I20250113 03:08:01.660807 24884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "118deb59b33541528dced43a0ac62bf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 15516 } } ignore_live_leader: true dest_uuid: "f00ce5a72049484a81629b9bc4bb358c"
I20250113 03:08:01.660809 24809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f936600153e46e59929870a2ec0e050" candidate_uuid: "118deb59b33541528dced43a0ac62bf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 15516 } } ignore_live_leader: true dest_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce"
I20250113 03:08:01.660889 24884 raft_consensus.cc:3049] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:08:01.660915 24809 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:01.660916 24884 raft_consensus.cc:738] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f00ce5a72049484a81629b9bc4bb358c, State: Running, Role: LEADER
I20250113 03:08:01.661008 24884 consensus_queue.cc:260] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15516, Committed index: 15516, Last appended: 1.15516, Last appended by leader: 15516, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:08:01.661089 24884 raft_consensus.cc:3054] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:01.661547 24809 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 118deb59b33541528dced43a0ac62bf4 in term 2.
I20250113 03:08:01.661685 24884 raft_consensus.cc:2463] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 118deb59b33541528dced43a0ac62bf4 in term 2.
I20250113 03:08:01.661700 24697 leader_election.cc:304] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 118deb59b33541528dced43a0ac62bf4, 7b7e45ba3ecc4721a0a5be32e15892ce; no voters:
I20250113 03:08:01.661877 24988 raft_consensus.cc:2798] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:08:01.661995 24988 raft_consensus.cc:695] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 LEADER]: Becoming Leader. State: Replica: 118deb59b33541528dced43a0ac62bf4, State: Running, Role: LEADER
I20250113 03:08:01.662045 24988 consensus_queue.cc:237] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15516, Committed index: 15516, Last appended: 1.15516, Last appended by leader: 15516, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } }
I20250113 03:08:01.662454 24627 catalog_manager.cc:5526] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 reported cstate change: term changed from 1 to 2, leader changed from f00ce5a72049484a81629b9bc4bb358c (127.20.18.67) to 118deb59b33541528dced43a0ac62bf4 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "118deb59b33541528dced43a0ac62bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7e45ba3ecc4721a0a5be32e15892ce" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f00ce5a72049484a81629b9bc4bb358c" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 44365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "118deb59b33541528dced43a0ac62bf4" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38401 } health_report { overall_health: HEALTHY } } }
I20250113 03:08:01.668365 20553 tablet_replica.cc:331] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: stopping tablet replica
I20250113 03:08:01.668427 20553 raft_consensus.cc:2238] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:08:01.668478 20553 pending_rounds.cc:62] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Trying to abort 1 pending ops.
I20250113 03:08:01.668504 20553 pending_rounds.cc:69] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Aborting op as it isn't in flight: id { term: 2 index: 15517 } timestamp: 7113677544088227840 op_type: NO_OP noop_request { }
I20250113 03:08:01.668565 20553 raft_consensus.cc:2883] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:08:01.668594 20553 raft_consensus.cc:2267] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:01.674110 20553 ts_tablet_manager.cc:1905] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 03:08:01.675488 20553 ts_tablet_manager.cc:1918] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15517
I20250113 03:08:01.675547 20553 log.cc:1198] T 5f936600153e46e59929870a2ec0e050 P 118deb59b33541528dced43a0ac62bf4: Deleting WAL directory at /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/5f936600153e46e59929870a2ec0e050
I20250113 03:08:01.683413 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:01.686000 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:01.697881 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:01.698869 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:01.701627 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:01.701721 20553 tablet_replica.cc:331] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce: stopping tablet replica
I20250113 03:08:01.701781 20553 raft_consensus.cc:2238] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:01.701846 20553 raft_consensus.cc:2267] T 5f936600153e46e59929870a2ec0e050 P 7b7e45ba3ecc4721a0a5be32e15892ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:01.715493 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:01.716235 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:01.718914 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:01.719002 20553 tablet_replica.cc:331] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c: stopping tablet replica
I20250113 03:08:01.719048 20553 raft_consensus.cc:2238] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:01.719090 20553 raft_consensus.cc:2267] T 5f936600153e46e59929870a2ec0e050 P f00ce5a72049484a81629b9bc4bb358c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:01.731688 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:01.732339 20553 master.cc:537] Master@127.20.18.126:36903 shutting down...
I20250113 03:08:01.734670 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:01.734728 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:01.734746 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P ba973badfeed4c9cb6f3c80e0c304287: stopping tablet replica
I20250113 03:08:01.746472 20553 master.cc:559] Master@127.20.18.126:36903 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotRunning (10828 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotFound
I20250113 03:08:01.760325 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:38779
I20250113 03:08:01.760466 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:01.761967 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.762636 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:01.763160 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.763639 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.763747 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:01.763774 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:01.763785 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737681763785 us; error 0 us; skew 500 ppm
I20250113 03:08:01.764302 20553 webserver.cc:458] Webserver started at http://127.20.18.126:45547/ using document root <none> and password file <none>
I20250113 03:08:01.764360 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:01.764389 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:01.764426 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:01.764726 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "bb84e89d8a1d4dba94899fdaf46979e0"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.765448 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:01.765798 25003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.765901 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.765937 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "bb84e89d8a1d4dba94899fdaf46979e0"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.765980 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:01.773185 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:01.773347 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:01.775548 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:38779
I20250113 03:08:01.778203 25055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:38779 every 8 connection(s)
I20250113 03:08:01.780616 25056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:01.781985 25056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0: Bootstrap starting.
I20250113 03:08:01.782275 25056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:01.782692 25056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0: No bootstrap required, opened a new log
I20250113 03:08:01.782792 25056 raft_consensus.cc:357] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER }
I20250113 03:08:01.782836 25056 raft_consensus.cc:383] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:01.782855 25056 raft_consensus.cc:738] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb84e89d8a1d4dba94899fdaf46979e0, State: Initialized, Role: FOLLOWER
I20250113 03:08:01.782908 25056 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER }
I20250113 03:08:01.782936 25056 raft_consensus.cc:397] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:01.782956 25056 raft_consensus.cc:491] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:01.782982 25056 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:01.783397 25056 raft_consensus.cc:513] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER }
I20250113 03:08:01.783449 25056 leader_election.cc:304] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb84e89d8a1d4dba94899fdaf46979e0; no voters:
I20250113 03:08:01.783537 25056 leader_election.cc:290] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:01.783576 25059 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:01.783656 25056 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:01.783696 25059 raft_consensus.cc:695] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 1 LEADER]: Becoming Leader. State: Replica: bb84e89d8a1d4dba94899fdaf46979e0, State: Running, Role: LEADER
I20250113 03:08:01.783746 25059 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER }
I20250113 03:08:01.783908 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb84e89d8a1d4dba94899fdaf46979e0. Latest consensus state: current_term: 1 leader_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER } }
I20250113 03:08:01.783934 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb84e89d8a1d4dba94899fdaf46979e0" member_type: VOTER } }
I20250113 03:08:01.784026 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:01.784060 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:01.784255 25065 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:01.784389 25065 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:01.784842 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:01.784966 25065 catalog_manager.cc:1348] Generated new cluster ID: 81258a1618504b0d8c2ecbdb1f6d9616
I20250113 03:08:01.785012 25065 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:01.806780 25065 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:01.807174 25065 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:01.812302 25065 catalog_manager.cc:5899] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0: Generated new TSK 0
I20250113 03:08:01.812368 25065 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:01.816550 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:01.817974 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.818598 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.819146 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:01.819624 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.819731 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:01.819769 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:01.819789 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737681819789 us; error 0 us; skew 500 ppm
I20250113 03:08:01.820395 20553 webserver.cc:458] Webserver started at http://127.20.18.65:44561/ using document root <none> and password file <none>
I20250113 03:08:01.820477 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:01.820518 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:01.820591 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:01.820850 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "e4cd93d46a8541f887a4e71c38f87305"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.821631 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:01.821995 25085 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.822106 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.822144 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "e4cd93d46a8541f887a4e71c38f87305"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.822183 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:01.830438 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:01.830590 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:01.830709 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:01.830871 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:01.830895 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.830921 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:01.830945 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.833422 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:38379
I20250113 03:08:01.833453 25148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:38379 every 8 connection(s)
I20250113 03:08:01.833731 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:01.835366 25153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.835538 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.836073 25156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.836666 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:01.836747 25149 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38779
I20250113 03:08:01.836808 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:08:01.836812 25149 heartbeater.cc:463] Registering TS with master...
W20250113 03:08:01.836853 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:01.836880 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737681836880 us; error 0 us; skew 500 ppm
I20250113 03:08:01.836947 25149 heartbeater.cc:510] Master 127.20.18.126:38779 requested a full tablet report, sending...
I20250113 03:08:01.837215 25020 ts_manager.cc:194] Registered new tserver with Master: e4cd93d46a8541f887a4e71c38f87305 (127.20.18.65:38379)
I20250113 03:08:01.837668 20553 webserver.cc:458] Webserver started at http://127.20.18.66:46641/ using document root <none> and password file <none>
I20250113 03:08:01.837745 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:01.837783 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:01.837827 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:01.837998 25020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60906
I20250113 03:08:01.838069 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "e543512d7d1b4964998b7ab87405f5ec"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.838918 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:01.839380 25161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.839490 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.839538 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "e543512d7d1b4964998b7ab87405f5ec"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.839577 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:01.878902 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:01.879097 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:01.879259 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:01.879449 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:01.879475 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.879495 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:01.879508 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.882122 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:42399
I20250113 03:08:01.882192 25224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:42399 every 8 connection(s)
I20250113 03:08:01.882540 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:01.883842 25228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.884474 25231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:01.885250 25229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:01.886009 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:01.886125 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:01.886153 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:01.886168 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737681886168 us; error 0 us; skew 500 ppm
I20250113 03:08:01.886184 25225 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38779
I20250113 03:08:01.886235 25225 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:01.886343 25225 heartbeater.cc:510] Master 127.20.18.126:38779 requested a full tablet report, sending...
I20250113 03:08:01.886770 25020 ts_manager.cc:194] Registered new tserver with Master: e543512d7d1b4964998b7ab87405f5ec (127.20.18.66:42399)
I20250113 03:08:01.886826 20553 webserver.cc:458] Webserver started at http://127.20.18.67:41383/ using document root <none> and password file <none>
I20250113 03:08:01.886910 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:01.886953 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:01.886996 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:01.887162 25020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60908
I20250113 03:08:01.887269 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "24f5652b6d8d48cba10ee6f6ed87844f"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.888058 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:01.888449 25236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.888569 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.888609 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "24f5652b6d8d48cba10ee6f6ed87844f"
format_stamp: "Formatted at 2025-01-13 03:08:01 on dist-test-slave-d4dl"
I20250113 03:08:01.888650 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:01.894974 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:01.895124 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:01.895249 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:01.895414 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:01.895438 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.895464 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:01.895489 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:01.898449 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:33745
I20250113 03:08:01.898496 25299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:33745 every 8 connection(s)
I20250113 03:08:01.900940 25300 heartbeater.cc:346] Connected to a master server at 127.20.18.126:38779
I20250113 03:08:01.900995 25300 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:01.901089 25300 heartbeater.cc:510] Master 127.20.18.126:38779 requested a full tablet report, sending...
I20250113 03:08:01.901278 25020 ts_manager.cc:194] Registered new tserver with Master: 24f5652b6d8d48cba10ee6f6ed87844f (127.20.18.67:33745)
I20250113 03:08:01.901579 25020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60912
I20250113 03:08:01.901895 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003280502s
I20250113 03:08:01.902174 20553 test_util.cc:274] Using random seed: -1063712162
I20250113 03:08:01.906090 25020 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60916:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:01.906208 25020 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:01.910374 25264 tablet_service.cc:1467] Processing CreateTablet for tablet cfdbba308479495295434d34a90f57c4 (DEFAULT_TABLE table=test-workload [id=44647baf56774c5d8bd5e085deda5222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:01.910509 25264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfdbba308479495295434d34a90f57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:01.910593 25189 tablet_service.cc:1467] Processing CreateTablet for tablet cfdbba308479495295434d34a90f57c4 (DEFAULT_TABLE table=test-workload [id=44647baf56774c5d8bd5e085deda5222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:01.910629 25113 tablet_service.cc:1467] Processing CreateTablet for tablet cfdbba308479495295434d34a90f57c4 (DEFAULT_TABLE table=test-workload [id=44647baf56774c5d8bd5e085deda5222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:01.910678 25189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfdbba308479495295434d34a90f57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:01.910754 25113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfdbba308479495295434d34a90f57c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:01.911916 25315 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: Bootstrap starting.
I20250113 03:08:01.912092 25316 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: Bootstrap starting.
I20250113 03:08:01.912379 25315 tablet_bootstrap.cc:654] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:01.912473 25316 tablet_bootstrap.cc:654] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:01.912818 25317 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: Bootstrap starting.
I20250113 03:08:01.912930 25315 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: No bootstrap required, opened a new log
I20250113 03:08:01.912971 25315 ts_tablet_manager.cc:1397] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:01.913090 25315 raft_consensus.cc:357] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.913137 25315 raft_consensus.cc:383] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:01.913151 25315 raft_consensus.cc:738] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cd93d46a8541f887a4e71c38f87305, State: Initialized, Role: FOLLOWER
I20250113 03:08:01.913188 25315 consensus_queue.cc:260] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.913187 25317 tablet_bootstrap.cc:654] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:01.913316 25316 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: No bootstrap required, opened a new log
I20250113 03:08:01.913362 25316 ts_tablet_manager.cc:1397] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:01.913376 25315 ts_tablet_manager.cc:1428] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 03:08:01.913461 25149 heartbeater.cc:502] Master 127.20.18.126:38779 was elected leader, sending a full tablet report...
I20250113 03:08:01.913522 25316 raft_consensus.cc:357] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.913578 25316 raft_consensus.cc:383] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:01.913601 25316 raft_consensus.cc:738] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e543512d7d1b4964998b7ab87405f5ec, State: Initialized, Role: FOLLOWER
I20250113 03:08:01.913655 25316 consensus_queue.cc:260] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.913812 25316 ts_tablet_manager.cc:1428] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:01.913877 25225 heartbeater.cc:502] Master 127.20.18.126:38779 was elected leader, sending a full tablet report...
I20250113 03:08:01.913944 25317 tablet_bootstrap.cc:492] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: No bootstrap required, opened a new log
I20250113 03:08:01.913985 25317 ts_tablet_manager.cc:1397] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:01.914137 25317 raft_consensus.cc:357] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.914273 25317 raft_consensus.cc:383] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:01.914294 25317 raft_consensus.cc:738] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f5652b6d8d48cba10ee6f6ed87844f, State: Initialized, Role: FOLLOWER
I20250113 03:08:01.914332 25317 consensus_queue.cc:260] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:01.914486 25317 ts_tablet_manager.cc:1428] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:01.914505 25300 heartbeater.cc:502] Master 127.20.18.126:38779 was elected leader, sending a full tablet report...
I20250113 03:08:02.033528 25323 raft_consensus.cc:491] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:02.033632 25323 raft_consensus.cc:513] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:02.033828 25323 leader_election.cc:290] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e543512d7d1b4964998b7ab87405f5ec (127.20.18.66:42399), e4cd93d46a8541f887a4e71c38f87305 (127.20.18.65:38379)
I20250113 03:08:02.036816 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdbba308479495295434d34a90f57c4" candidate_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e543512d7d1b4964998b7ab87405f5ec" is_pre_election: true
I20250113 03:08:02.036818 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdbba308479495295434d34a90f57c4" candidate_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cd93d46a8541f887a4e71c38f87305" is_pre_election: true
I20250113 03:08:02.036914 25199 raft_consensus.cc:2463] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24f5652b6d8d48cba10ee6f6ed87844f in term 0.
I20250113 03:08:02.036926 25123 raft_consensus.cc:2463] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24f5652b6d8d48cba10ee6f6ed87844f in term 0.
I20250113 03:08:02.037106 25238 leader_election.cc:304] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f5652b6d8d48cba10ee6f6ed87844f, e543512d7d1b4964998b7ab87405f5ec; no voters:
I20250113 03:08:02.037185 25323 raft_consensus.cc:2798] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:02.037225 25323 raft_consensus.cc:491] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:02.037238 25323 raft_consensus.cc:3054] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:02.037806 25323 raft_consensus.cc:513] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:02.037914 25323 leader_election.cc:290] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [CANDIDATE]: Term 1 election: Requested vote from peers e543512d7d1b4964998b7ab87405f5ec (127.20.18.66:42399), e4cd93d46a8541f887a4e71c38f87305 (127.20.18.65:38379)
I20250113 03:08:02.038049 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdbba308479495295434d34a90f57c4" candidate_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e543512d7d1b4964998b7ab87405f5ec"
I20250113 03:08:02.038062 25123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdbba308479495295434d34a90f57c4" candidate_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cd93d46a8541f887a4e71c38f87305"
I20250113 03:08:02.038105 25199 raft_consensus.cc:3054] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:02.038110 25123 raft_consensus.cc:3054] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:02.038710 25123 raft_consensus.cc:2463] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24f5652b6d8d48cba10ee6f6ed87844f in term 1.
I20250113 03:08:02.038715 25199 raft_consensus.cc:2463] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24f5652b6d8d48cba10ee6f6ed87844f in term 1.
I20250113 03:08:02.038843 25238 leader_election.cc:304] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24f5652b6d8d48cba10ee6f6ed87844f, e543512d7d1b4964998b7ab87405f5ec; no voters:
I20250113 03:08:02.038915 25323 raft_consensus.cc:2798] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:02.039038 25323 raft_consensus.cc:695] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 1 LEADER]: Becoming Leader. State: Replica: 24f5652b6d8d48cba10ee6f6ed87844f, State: Running, Role: LEADER
I20250113 03:08:02.039084 25323 consensus_queue.cc:237] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } }
I20250113 03:08:02.039577 25019 catalog_manager.cc:5526] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f reported cstate change: term changed from 0 to 1, leader changed from <none> to 24f5652b6d8d48cba10ee6f6ed87844f (127.20.18.67). New cstate: current_term: 1 leader_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24f5652b6d8d48cba10ee6f6ed87844f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33745 } health_report { overall_health: HEALTHY } } }
I20250113 03:08:02.058249 25123 raft_consensus.cc:1270] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 1 FOLLOWER]: Refusing update from remote peer 24f5652b6d8d48cba10ee6f6ed87844f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:02.058249 25199 raft_consensus.cc:1270] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 1 FOLLOWER]: Refusing update from remote peer 24f5652b6d8d48cba10ee6f6ed87844f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:02.058456 25325 consensus_queue.cc:1035] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cd93d46a8541f887a4e71c38f87305" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:02.058543 25323 consensus_queue.cc:1035] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e543512d7d1b4964998b7ab87405f5ec" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:12.060706 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:12.062944 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:12.063058 20553 tablet_replica.cc:331] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305: stopping tablet replica
I20250113 03:08:12.063112 20553 raft_consensus.cc:2238] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:12.063161 20553 raft_consensus.cc:2267] T cfdbba308479495295434d34a90f57c4 P e4cd93d46a8541f887a4e71c38f87305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:12.068475 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:12.069129 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:12.071300 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:12.071377 20553 tablet_replica.cc:331] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec: stopping tablet replica
I20250113 03:08:12.071422 20553 raft_consensus.cc:2238] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:12.071471 20553 raft_consensus.cc:2267] T cfdbba308479495295434d34a90f57c4 P e543512d7d1b4964998b7ab87405f5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:12.074558 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:12.075145 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:12.077255 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:12.077332 20553 tablet_replica.cc:331] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f: stopping tablet replica
I20250113 03:08:12.077383 20553 raft_consensus.cc:2238] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:12.077431 20553 raft_consensus.cc:2267] T cfdbba308479495295434d34a90f57c4 P 24f5652b6d8d48cba10ee6f6ed87844f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:12.099382 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:12.100157 20553 master.cc:537] Master@127.20.18.126:38779 shutting down...
I20250113 03:08:12.103873 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:12.103930 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:12.103962 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P bb84e89d8a1d4dba94899fdaf46979e0: stopping tablet replica
I20250113 03:08:12.115530 20553 master.cc:559] Master@127.20.18.126:38779 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotFound (10373 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientGetMetadata
I20250113 03:08:12.133881 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:33189
I20250113 03:08:12.134029 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:12.135952 25371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.136173 25372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.136951 25374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:12.137667 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:12.137761 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:12.137799 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:12.137818 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737692137818 us; error 0 us; skew 500 ppm
I20250113 03:08:12.138418 20553 webserver.cc:458] Webserver started at http://127.20.18.126:45481/ using document root <none> and password file <none>
I20250113 03:08:12.138490 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:12.138535 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:12.138576 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:12.138823 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "b0b9857cc43b4d0f91197e93814050e4"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.139571 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:12.139938 25379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.140046 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.140084 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "b0b9857cc43b4d0f91197e93814050e4"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.140134 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:12.158612 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:12.158803 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:12.161588 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:33189
I20250113 03:08:12.168273 25431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:33189 every 8 connection(s)
I20250113 03:08:12.170207 25432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:12.171177 25432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4: Bootstrap starting.
I20250113 03:08:12.171489 25432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:12.171916 25432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4: No bootstrap required, opened a new log
I20250113 03:08:12.172019 25432 raft_consensus.cc:357] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER }
I20250113 03:08:12.172066 25432 raft_consensus.cc:383] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:12.172078 25432 raft_consensus.cc:738] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b9857cc43b4d0f91197e93814050e4, State: Initialized, Role: FOLLOWER
I20250113 03:08:12.172122 25432 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER }
I20250113 03:08:12.172154 25432 raft_consensus.cc:397] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:12.172178 25432 raft_consensus.cc:491] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:12.172201 25432 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:12.172678 25432 raft_consensus.cc:513] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER }
I20250113 03:08:12.172729 25432 leader_election.cc:304] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b9857cc43b4d0f91197e93814050e4; no voters:
I20250113 03:08:12.172818 25432 leader_election.cc:290] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:12.172869 25435 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:12.172971 25432 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:12.173012 25435 raft_consensus.cc:695] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 1 LEADER]: Becoming Leader. State: Replica: b0b9857cc43b4d0f91197e93814050e4, State: Running, Role: LEADER
I20250113 03:08:12.173058 25435 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER }
I20250113 03:08:12.173275 25436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b9857cc43b4d0f91197e93814050e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER } }
I20250113 03:08:12.173331 25436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:12.173297 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b9857cc43b4d0f91197e93814050e4. Latest consensus state: current_term: 1 leader_uuid: "b0b9857cc43b4d0f91197e93814050e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b9857cc43b4d0f91197e93814050e4" member_type: VOTER } }
I20250113 03:08:12.173362 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:12.173532 25440 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:12.173660 25440 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:12.174144 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:12.174156 25440 catalog_manager.cc:1348] Generated new cluster ID: 36bd093b0a184b26bc2ce82fe38f152a
I20250113 03:08:12.174208 25440 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:12.187918 25440 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:12.188287 25440 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:12.192188 25440 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4: Generated new TSK 0
I20250113 03:08:12.192248 25440 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:12.205951 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:12.207356 25453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.208168 25454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.208894 25456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:12.209426 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:12.209491 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:12.209512 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:12.209523 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737692209524 us; error 0 us; skew 500 ppm
I20250113 03:08:12.210114 20553 webserver.cc:458] Webserver started at http://127.20.18.65:34261/ using document root <none> and password file <none>
I20250113 03:08:12.210180 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:12.210211 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:12.210247 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:12.210475 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "ceb09b852d7a4b01b517859096d775ce"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.211212 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:12.211585 25461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.211694 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.211738 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "ceb09b852d7a4b01b517859096d775ce"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.211771 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:12.229034 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:12.229221 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:12.229377 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:12.229548 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:12.229573 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.229593 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:12.229617 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.232093 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:33231
I20250113 03:08:12.232295 25524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:33231 every 8 connection(s)
I20250113 03:08:12.232517 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:12.233716 25529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.234478 25530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:12.235239 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:12.235350 25525 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33189
I20250113 03:08:12.235390 25525 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:12.235482 25525 heartbeater.cc:510] Master 127.20.18.126:33189 requested a full tablet report, sending...
I20250113 03:08:12.235702 25393 ts_manager.cc:194] Registered new tserver with Master: ceb09b852d7a4b01b517859096d775ce (127.20.18.65:33231)
W20250113 03:08:12.235816 25532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:12.235954 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:12.235986 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:12.235998 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737692235999 us; error 0 us; skew 500 ppm
I20250113 03:08:12.236327 25393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40266
I20250113 03:08:12.236711 20553 webserver.cc:458] Webserver started at http://127.20.18.66:40785/ using document root <none> and password file <none>
I20250113 03:08:12.236802 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:12.236848 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:12.236882 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:12.237135 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "7d879f0e5f9343218fed455b8c24462e"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.237931 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:12.238297 25537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.238415 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.238459 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "7d879f0e5f9343218fed455b8c24462e"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.238492 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:12.242579 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:12.242710 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:12.242806 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:12.242944 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:12.242964 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.242987 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:12.243002 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.245594 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:35777
I20250113 03:08:12.245640 25600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:35777 every 8 connection(s)
I20250113 03:08:12.245937 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:12.247524 25604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.247737 25605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:12.248325 25607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:12.248764 25601 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33189
I20250113 03:08:12.248821 25601 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:12.248955 25601 heartbeater.cc:510] Master 127.20.18.126:33189 requested a full tablet report, sending...
I20250113 03:08:12.249071 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:12.249163 25393 ts_manager.cc:194] Registered new tserver with Master: 7d879f0e5f9343218fed455b8c24462e (127.20.18.66:35777)
I20250113 03:08:12.249226 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:12.249262 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:12.249290 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737692249290 us; error 0 us; skew 500 ppm
I20250113 03:08:12.249527 25393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40282
I20250113 03:08:12.249965 20553 webserver.cc:458] Webserver started at http://127.20.18.67:45071/ using document root <none> and password file <none>
I20250113 03:08:12.250042 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:12.250093 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:12.250139 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:12.250370 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "9f3c7108f941485ba54e443af06a988a"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.251149 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:12.251552 25612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.251659 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.251716 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "9f3c7108f941485ba54e443af06a988a"
format_stamp: "Formatted at 2025-01-13 03:08:12 on dist-test-slave-d4dl"
I20250113 03:08:12.251762 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:12.260953 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:12.261153 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:12.261284 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:12.261453 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:12.261476 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.261507 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:12.261533 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:12.264356 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:40495
I20250113 03:08:12.264408 25675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:40495 every 8 connection(s)
I20250113 03:08:12.267036 25676 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33189
I20250113 03:08:12.267095 25676 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:12.267190 25676 heartbeater.cc:510] Master 127.20.18.126:33189 requested a full tablet report, sending...
I20250113 03:08:12.267400 25393 ts_manager.cc:194] Registered new tserver with Master: 9f3c7108f941485ba54e443af06a988a (127.20.18.67:40495)
I20250113 03:08:12.267699 25393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40284
I20250113 03:08:12.267863 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003313748s
I20250113 03:08:12.268177 20553 test_util.cc:274] Using random seed: -1053346160
I20250113 03:08:12.272089 25393 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40286:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:12.272241 25393 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:12.276492 25565 tablet_service.cc:1467] Processing CreateTablet for tablet f36e85129b774100917d51cc1b297f41 (DEFAULT_TABLE table=test-workload [id=1babcc6d432d4c719a0709618cb9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:12.276644 25565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36e85129b774100917d51cc1b297f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:12.276644 25640 tablet_service.cc:1467] Processing CreateTablet for tablet f36e85129b774100917d51cc1b297f41 (DEFAULT_TABLE table=test-workload [id=1babcc6d432d4c719a0709618cb9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:12.276729 25489 tablet_service.cc:1467] Processing CreateTablet for tablet f36e85129b774100917d51cc1b297f41 (DEFAULT_TABLE table=test-workload [id=1babcc6d432d4c719a0709618cb9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:12.276755 25640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36e85129b774100917d51cc1b297f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:12.276829 25489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36e85129b774100917d51cc1b297f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:12.278062 25691 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: Bootstrap starting.
I20250113 03:08:12.278432 25691 tablet_bootstrap.cc:654] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:12.278714 25692 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: Bootstrap starting.
I20250113 03:08:12.278892 25694 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: Bootstrap starting.
I20250113 03:08:12.279222 25692 tablet_bootstrap.cc:654] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:12.279295 25694 tablet_bootstrap.cc:654] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:12.279670 25691 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: No bootstrap required, opened a new log
I20250113 03:08:12.279716 25691 ts_tablet_manager.cc:1397] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:08:12.279819 25694 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: No bootstrap required, opened a new log
I20250113 03:08:12.279855 25694 ts_tablet_manager.cc:1397] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:12.279865 25691 raft_consensus.cc:357] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.279937 25691 raft_consensus.cc:383] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:12.279961 25691 raft_consensus.cc:738] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3c7108f941485ba54e443af06a988a, State: Initialized, Role: FOLLOWER
I20250113 03:08:12.279976 25694 raft_consensus.cc:357] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.280025 25694 raft_consensus.cc:383] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:12.280058 25694 raft_consensus.cc:738] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb09b852d7a4b01b517859096d775ce, State: Initialized, Role: FOLLOWER
I20250113 03:08:12.280030 25691 consensus_queue.cc:260] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.280115 25694 consensus_queue.cc:260] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.280210 25691 ts_tablet_manager.cc:1428] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:12.280236 25676 heartbeater.cc:502] Master 127.20.18.126:33189 was elected leader, sending a full tablet report...
I20250113 03:08:12.280313 25525 heartbeater.cc:502] Master 127.20.18.126:33189 was elected leader, sending a full tablet report...
I20250113 03:08:12.280391 25692 tablet_bootstrap.cc:492] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: No bootstrap required, opened a new log
I20250113 03:08:12.280424 25692 ts_tablet_manager.cc:1397] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:08:12.280580 25692 raft_consensus.cc:357] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.280655 25692 raft_consensus.cc:383] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:12.280676 25692 raft_consensus.cc:738] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d879f0e5f9343218fed455b8c24462e, State: Initialized, Role: FOLLOWER
I20250113 03:08:12.280717 25692 consensus_queue.cc:260] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.280853 25692 ts_tablet_manager.cc:1428] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:12.280251 25694 ts_tablet_manager.cc:1428] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:12.280906 25601 heartbeater.cc:502] Master 127.20.18.126:33189 was elected leader, sending a full tablet report...
I20250113 03:08:12.391673 25699 raft_consensus.cc:491] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:12.391765 25699 raft_consensus.cc:513] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.391957 25699 leader_election.cc:290] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb09b852d7a4b01b517859096d775ce (127.20.18.65:33231), 9f3c7108f941485ba54e443af06a988a (127.20.18.67:40495)
I20250113 03:08:12.394327 25499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "7d879f0e5f9343218fed455b8c24462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb09b852d7a4b01b517859096d775ce" is_pre_election: true
I20250113 03:08:12.394330 25650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "7d879f0e5f9343218fed455b8c24462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3c7108f941485ba54e443af06a988a" is_pre_election: true
I20250113 03:08:12.394429 25499 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d879f0e5f9343218fed455b8c24462e in term 0.
I20250113 03:08:12.394447 25650 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d879f0e5f9343218fed455b8c24462e in term 0.
I20250113 03:08:12.394562 25541 leader_election.cc:304] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d879f0e5f9343218fed455b8c24462e, ceb09b852d7a4b01b517859096d775ce; no voters:
I20250113 03:08:12.394636 25699 raft_consensus.cc:2798] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:12.394671 25699 raft_consensus.cc:491] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:12.394691 25699 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:12.395220 25699 raft_consensus.cc:513] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.395319 25699 leader_election.cc:290] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [CANDIDATE]: Term 1 election: Requested vote from peers ceb09b852d7a4b01b517859096d775ce (127.20.18.65:33231), 9f3c7108f941485ba54e443af06a988a (127.20.18.67:40495)
I20250113 03:08:12.395450 25499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "7d879f0e5f9343218fed455b8c24462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb09b852d7a4b01b517859096d775ce"
I20250113 03:08:12.395501 25499 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:12.395463 25650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "7d879f0e5f9343218fed455b8c24462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3c7108f941485ba54e443af06a988a"
I20250113 03:08:12.395534 25650 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:12.396055 25499 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d879f0e5f9343218fed455b8c24462e in term 1.
I20250113 03:08:12.396150 25650 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d879f0e5f9343218fed455b8c24462e in term 1.
I20250113 03:08:12.396248 25541 leader_election.cc:304] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d879f0e5f9343218fed455b8c24462e, ceb09b852d7a4b01b517859096d775ce; no voters:
I20250113 03:08:12.396354 25699 raft_consensus.cc:2798] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:12.396485 25699 raft_consensus.cc:695] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 LEADER]: Becoming Leader. State: Replica: 7d879f0e5f9343218fed455b8c24462e, State: Running, Role: LEADER
I20250113 03:08:12.396566 25699 consensus_queue.cc:237] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:12.397065 25396 catalog_manager.cc:5526] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d879f0e5f9343218fed455b8c24462e (127.20.18.66). New cstate: current_term: 1 leader_uuid: "7d879f0e5f9343218fed455b8c24462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:12.430662 25650 raft_consensus.cc:1270] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 1 FOLLOWER]: Refusing update from remote peer 7d879f0e5f9343218fed455b8c24462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:12.430662 25499 raft_consensus.cc:1270] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 1 FOLLOWER]: Refusing update from remote peer 7d879f0e5f9343218fed455b8c24462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:12.430907 25701 consensus_queue.cc:1035] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:12.430987 25699 consensus_queue.cc:1035] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:22.966488 25734 raft_consensus.cc:988] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: : Instructing follower ceb09b852d7a4b01b517859096d775ce to start an election
I20250113 03:08:22.966559 25734 raft_consensus.cc:1076] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 LEADER]: Signalling peer ceb09b852d7a4b01b517859096d775ce to start an election
I20250113 03:08:22.966780 25499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f36e85129b774100917d51cc1b297f41"
dest_uuid: "ceb09b852d7a4b01b517859096d775ce"
from {username='slave'} at 127.0.0.1:50428
I20250113 03:08:22.966847 25499 raft_consensus.cc:491] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:08:22.966866 25499 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:22.967442 25499 raft_consensus.cc:513] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:22.967567 25499 leader_election.cc:290] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [CANDIDATE]: Term 2 election: Requested vote from peers 7d879f0e5f9343218fed455b8c24462e (127.20.18.66:35777), 9f3c7108f941485ba54e443af06a988a (127.20.18.67:40495)
I20250113 03:08:22.969938 25575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "ceb09b852d7a4b01b517859096d775ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 15494 } } ignore_live_leader: true dest_uuid: "7d879f0e5f9343218fed455b8c24462e"
I20250113 03:08:22.969944 25650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36e85129b774100917d51cc1b297f41" candidate_uuid: "ceb09b852d7a4b01b517859096d775ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 15494 } } ignore_live_leader: true dest_uuid: "9f3c7108f941485ba54e443af06a988a"
I20250113 03:08:22.970012 25575 raft_consensus.cc:3049] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:08:22.970062 25650 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:22.970070 25575 raft_consensus.cc:738] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d879f0e5f9343218fed455b8c24462e, State: Running, Role: LEADER
I20250113 03:08:22.970144 25575 consensus_queue.cc:260] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15494, Committed index: 15494, Last appended: 1.15494, Last appended by leader: 15494, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:22.970217 25575 raft_consensus.cc:3054] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:22.970665 25650 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb09b852d7a4b01b517859096d775ce in term 2.
I20250113 03:08:22.970806 25575 raft_consensus.cc:2463] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb09b852d7a4b01b517859096d775ce in term 2.
I20250113 03:08:22.970832 25463 leader_election.cc:304] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f3c7108f941485ba54e443af06a988a, ceb09b852d7a4b01b517859096d775ce; no voters:
I20250113 03:08:22.970999 25754 raft_consensus.cc:2798] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:08:22.971115 25754 raft_consensus.cc:695] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 2 LEADER]: Becoming Leader. State: Replica: ceb09b852d7a4b01b517859096d775ce, State: Running, Role: LEADER
I20250113 03:08:22.971160 25754 consensus_queue.cc:237] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15494, Committed index: 15494, Last appended: 1.15494, Last appended by leader: 15494, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } }
I20250113 03:08:22.971576 25389 catalog_manager.cc:5526] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce reported cstate change: term changed from 1 to 2, leader changed from 7d879f0e5f9343218fed455b8c24462e (127.20.18.66) to ceb09b852d7a4b01b517859096d775ce (127.20.18.65). New cstate: current_term: 2 leader_uuid: "ceb09b852d7a4b01b517859096d775ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb09b852d7a4b01b517859096d775ce" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 33231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:22.977931 25650 raft_consensus.cc:1270] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 2 FOLLOWER]: Refusing update from remote peer ceb09b852d7a4b01b517859096d775ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 15494. Preceding OpId from leader: term: 2 index: 15496. (index mismatch)
I20250113 03:08:22.977945 25575 raft_consensus.cc:1270] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 2 FOLLOWER]: Refusing update from remote peer ceb09b852d7a4b01b517859096d775ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 15494. Preceding OpId from leader: term: 2 index: 15496. (index mismatch)
I20250113 03:08:22.978142 25755 consensus_queue.cc:1035] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d879f0e5f9343218fed455b8c24462e" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15495, Last known committed idx: 15494, Time since last communication: 0.000s
I20250113 03:08:22.978214 25754 consensus_queue.cc:1035] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3c7108f941485ba54e443af06a988a" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15495, Last known committed idx: 15494, Time since last communication: 0.000s
I20250113 03:08:22.983601 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:22.985975 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:22.986078 20553 tablet_replica.cc:331] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce: stopping tablet replica
I20250113 03:08:22.986127 20553 raft_consensus.cc:2238] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:08:22.986172 20553 raft_consensus.cc:2267] T f36e85129b774100917d51cc1b297f41 P ceb09b852d7a4b01b517859096d775ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:22.999833 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:23.000654 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:23.003906 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:23.003994 20553 tablet_replica.cc:331] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e: stopping tablet replica
I20250113 03:08:23.004047 20553 raft_consensus.cc:2238] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:23.004086 20553 raft_consensus.cc:2267] T f36e85129b774100917d51cc1b297f41 P 7d879f0e5f9343218fed455b8c24462e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:23.015426 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:23.016111 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:23.019273 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:23.019385 20553 tablet_replica.cc:331] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a: stopping tablet replica
I20250113 03:08:23.019438 20553 raft_consensus.cc:2238] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:23.019481 20553 raft_consensus.cc:2267] T f36e85129b774100917d51cc1b297f41 P 9f3c7108f941485ba54e443af06a988a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:23.032810 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:23.033485 20553 master.cc:537] Master@127.20.18.126:33189 shutting down...
I20250113 03:08:23.035879 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:23.035943 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:23.035988 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P b0b9857cc43b4d0f91197e93814050e4: stopping tablet replica
I20250113 03:08:23.047663 20553 master.cc:559] Master@127.20.18.126:33189 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientGetMetadata (10932 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock
I20250113 03:08:23.066020 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:43469
I20250113 03:08:23.066175 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:23.068094 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.068255 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.069124 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:23.069814 25773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.069933 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:23.069963 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:23.069984 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737703069984 us; error 0 us; skew 500 ppm
I20250113 03:08:23.070586 20553 webserver.cc:458] Webserver started at http://127.20.18.126:39257/ using document root <none> and password file <none>
I20250113 03:08:23.070649 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:23.070681 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:23.070715 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:23.070950 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "213b2706a6df443ba0e4707e5d7f4036"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.071717 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.072110 25780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.072245 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.072296 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "213b2706a6df443ba0e4707e5d7f4036"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.072340 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:23.082909 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:23.083076 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:23.085220 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:43469
I20250113 03:08:23.085688 25833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:23.090492 25832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:43469 every 8 connection(s)
I20250113 03:08:23.095382 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036: Bootstrap starting.
I20250113 03:08:23.095635 25833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:23.096027 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036: No bootstrap required, opened a new log
I20250113 03:08:23.096119 25833 raft_consensus.cc:357] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER }
I20250113 03:08:23.096158 25833 raft_consensus.cc:383] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:23.096170 25833 raft_consensus.cc:738] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b2706a6df443ba0e4707e5d7f4036, State: Initialized, Role: FOLLOWER
I20250113 03:08:23.096201 25833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER }
I20250113 03:08:23.096242 25833 raft_consensus.cc:397] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:23.096263 25833 raft_consensus.cc:491] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:23.096282 25833 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:23.096747 25833 raft_consensus.cc:513] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER }
I20250113 03:08:23.096798 25833 leader_election.cc:304] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213b2706a6df443ba0e4707e5d7f4036; no voters:
I20250113 03:08:23.096871 25833 leader_election.cc:290] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:23.096928 25836 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:23.097004 25833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:23.097056 25836 raft_consensus.cc:695] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 1 LEADER]: Becoming Leader. State: Replica: 213b2706a6df443ba0e4707e5d7f4036, State: Running, Role: LEADER
I20250113 03:08:23.097108 25836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER }
I20250113 03:08:23.097326 25837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "213b2706a6df443ba0e4707e5d7f4036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER } }
I20250113 03:08:23.097335 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 213b2706a6df443ba0e4707e5d7f4036. Latest consensus state: current_term: 1 leader_uuid: "213b2706a6df443ba0e4707e5d7f4036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213b2706a6df443ba0e4707e5d7f4036" member_type: VOTER } }
I20250113 03:08:23.097492 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:23.097477 25837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:23.097699 25845 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:23.097810 25845 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:23.098148 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:23.098258 25845 catalog_manager.cc:1348] Generated new cluster ID: cd53f083d11a49c28b44218bffbbc39c
I20250113 03:08:23.098304 25845 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:23.100922 25845 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:23.101290 25845 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:23.106748 25845 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036: Generated new TSK 0
I20250113 03:08:23.106810 25845 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:23.113693 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:23.114965 25854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.115707 25855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.116315 25857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.116950 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:23.117031 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:23.117054 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:23.117066 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737703117067 us; error 0 us; skew 500 ppm
I20250113 03:08:23.117611 20553 webserver.cc:458] Webserver started at http://127.20.18.65:35533/ using document root <none> and password file <none>
I20250113 03:08:23.117673 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:23.117704 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:23.117735 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:23.117944 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "c0cffa3d3db2478e954fcd9dcc84e093"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.118670 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.119035 25862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.119129 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.119160 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "c0cffa3d3db2478e954fcd9dcc84e093"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.119197 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:23.130610 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:23.130753 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:23.130872 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:23.131031 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:23.131052 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.131074 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:23.131089 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.133615 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:36325
I20250113 03:08:23.134117 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:08:23.134130 25925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:36325 every 8 connection(s)
W20250113 03:08:23.135350 25928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.135914 25933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.136587 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:23.137301 25930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.137372 25926 heartbeater.cc:346] Connected to a master server at 127.20.18.126:43469
I20250113 03:08:23.137420 25926 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:23.137490 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:08:23.137521 25926 heartbeater.cc:510] Master 127.20.18.126:43469 requested a full tablet report, sending...
W20250113 03:08:23.137530 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:23.137574 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737703137574 us; error 0 us; skew 500 ppm
I20250113 03:08:23.137744 25797 ts_manager.cc:194] Registered new tserver with Master: c0cffa3d3db2478e954fcd9dcc84e093 (127.20.18.65:36325)
I20250113 03:08:23.138237 20553 webserver.cc:458] Webserver started at http://127.20.18.66:36613/ using document root <none> and password file <none>
I20250113 03:08:23.138327 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:23.138365 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:23.138402 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:23.138422 25797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60220
I20250113 03:08:23.138674 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "df69f0ce60774a99874c1655d3049208"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.139456 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.139822 25938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.139935 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.139976 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "df69f0ce60774a99874c1655d3049208"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.140015 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:23.151777 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:23.151897 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:23.151983 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:23.152108 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:23.152128 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.152145 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:23.152160 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.154523 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:39215
I20250113 03:08:23.154573 26001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:39215 every 8 connection(s)
I20250113 03:08:23.154807 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:23.155987 26005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.156774 26006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:23.157398 26008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:23.157584 26002 heartbeater.cc:346] Connected to a master server at 127.20.18.126:43469
I20250113 03:08:23.157632 26002 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:23.157724 26002 heartbeater.cc:510] Master 127.20.18.126:43469 requested a full tablet report, sending...
I20250113 03:08:23.157915 25797 ts_manager.cc:194] Registered new tserver with Master: df69f0ce60774a99874c1655d3049208 (127.20.18.66:39215)
I20250113 03:08:23.158041 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:23.158160 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:23.158188 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:23.158203 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737703158204 us; error 0 us; skew 500 ppm
I20250113 03:08:23.158365 25797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60228
I20250113 03:08:23.158752 20553 webserver.cc:458] Webserver started at http://127.20.18.67:35881/ using document root <none> and password file <none>
I20250113 03:08:23.158814 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:23.158856 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:23.158897 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:23.159118 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "d6f9e7242bfb4433887d936becc85a7f"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.159862 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:23.160239 26013 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.160357 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.160391 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "d6f9e7242bfb4433887d936becc85a7f"
format_stamp: "Formatted at 2025-01-13 03:08:23 on dist-test-slave-d4dl"
I20250113 03:08:23.160425 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:23.164479 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:23.164641 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:23.164736 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:23.164885 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:23.164908 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.164937 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:23.164963 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.167845 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:38267
I20250113 03:08:23.167896 26076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:38267 every 8 connection(s)
I20250113 03:08:23.170136 26077 heartbeater.cc:346] Connected to a master server at 127.20.18.126:43469
I20250113 03:08:23.170178 26077 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:23.170257 26077 heartbeater.cc:510] Master 127.20.18.126:43469 requested a full tablet report, sending...
I20250113 03:08:23.170444 25797 ts_manager.cc:194] Registered new tserver with Master: d6f9e7242bfb4433887d936becc85a7f (127.20.18.67:38267)
I20250113 03:08:23.170735 25797 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60238
I20250113 03:08:23.171257 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003252858s
I20250113 03:08:23.171521 20553 test_util.cc:274] Using random seed: -1042442815
I20250113 03:08:23.175443 25797 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60242:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:23.175554 25797 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:23.179487 25890 tablet_service.cc:1467] Processing CreateTablet for tablet b1c39799ec9e43e7ae1e4a3e3c26480f (DEFAULT_TABLE table=test-workload [id=e4c7c3beb42a466f87d2286a6bf422a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:23.179574 26041 tablet_service.cc:1467] Processing CreateTablet for tablet b1c39799ec9e43e7ae1e4a3e3c26480f (DEFAULT_TABLE table=test-workload [id=e4c7c3beb42a466f87d2286a6bf422a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:23.179594 25890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c39799ec9e43e7ae1e4a3e3c26480f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:23.179603 25966 tablet_service.cc:1467] Processing CreateTablet for tablet b1c39799ec9e43e7ae1e4a3e3c26480f (DEFAULT_TABLE table=test-workload [id=e4c7c3beb42a466f87d2286a6bf422a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:23.179721 25966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c39799ec9e43e7ae1e4a3e3c26480f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:23.179639 26041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1c39799ec9e43e7ae1e4a3e3c26480f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:23.180811 26093 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: Bootstrap starting.
I20250113 03:08:23.181252 26093 tablet_bootstrap.cc:654] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:23.181790 26093 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: No bootstrap required, opened a new log
I20250113 03:08:23.181838 26093 ts_tablet_manager.cc:1397] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.181917 26095 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: Bootstrap starting.
I20250113 03:08:23.181944 26092 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: Bootstrap starting.
I20250113 03:08:23.182034 26093 raft_consensus.cc:357] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.182104 26093 raft_consensus.cc:383] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:23.182126 26093 raft_consensus.cc:738] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df69f0ce60774a99874c1655d3049208, State: Initialized, Role: FOLLOWER
I20250113 03:08:23.182188 26093 consensus_queue.cc:260] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.182319 26095 tablet_bootstrap.cc:654] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:23.182365 26002 heartbeater.cc:502] Master 127.20.18.126:43469 was elected leader, sending a full tablet report...
I20250113 03:08:23.182328 26093 ts_tablet_manager.cc:1428] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.182328 26092 tablet_bootstrap.cc:654] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:23.182934 26095 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: No bootstrap required, opened a new log
I20250113 03:08:23.182981 26095 ts_tablet_manager.cc:1397] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.182989 26092 tablet_bootstrap.cc:492] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: No bootstrap required, opened a new log
I20250113 03:08:23.183044 26092 ts_tablet_manager.cc:1397] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:23.183118 26095 raft_consensus.cc:357] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.183161 26092 raft_consensus.cc:357] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.183161 26095 raft_consensus.cc:383] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:23.183210 26092 raft_consensus.cc:383] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:23.183240 26095 raft_consensus.cc:738] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f9e7242bfb4433887d936becc85a7f, State: Initialized, Role: FOLLOWER
I20250113 03:08:23.183241 26092 raft_consensus.cc:738] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cffa3d3db2478e954fcd9dcc84e093, State: Initialized, Role: FOLLOWER
I20250113 03:08:23.183321 26095 consensus_queue.cc:260] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.183324 26092 consensus_queue.cc:260] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.183501 26077 heartbeater.cc:502] Master 127.20.18.126:43469 was elected leader, sending a full tablet report...
I20250113 03:08:23.183483 26095 ts_tablet_manager.cc:1428] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:23.183486 26092 ts_tablet_manager.cc:1428] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:23.183511 25926 heartbeater.cc:502] Master 127.20.18.126:43469 was elected leader, sending a full tablet report...
I20250113 03:08:23.316510 26096 raft_consensus.cc:491] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:23.316623 26096 raft_consensus.cc:513] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.316790 26096 leader_election.cc:290] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cffa3d3db2478e954fcd9dcc84e093 (127.20.18.65:36325), d6f9e7242bfb4433887d936becc85a7f (127.20.18.67:38267)
I20250113 03:08:23.319808 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "df69f0ce60774a99874c1655d3049208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f9e7242bfb4433887d936becc85a7f" is_pre_election: true
I20250113 03:08:23.319808 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "df69f0ce60774a99874c1655d3049208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" is_pre_election: true
I20250113 03:08:23.319962 26051 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f0ce60774a99874c1655d3049208 in term 0.
I20250113 03:08:23.319967 25900 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df69f0ce60774a99874c1655d3049208 in term 0.
I20250113 03:08:23.320151 25941 leader_election.cc:304] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6f9e7242bfb4433887d936becc85a7f, df69f0ce60774a99874c1655d3049208; no voters:
I20250113 03:08:23.320286 26096 raft_consensus.cc:2798] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:23.320319 26096 raft_consensus.cc:491] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:23.320333 26096 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:23.320905 26096 raft_consensus.cc:513] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.321010 26096 leader_election.cc:290] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [CANDIDATE]: Term 1 election: Requested vote from peers c0cffa3d3db2478e954fcd9dcc84e093 (127.20.18.65:36325), d6f9e7242bfb4433887d936becc85a7f (127.20.18.67:38267)
I20250113 03:08:23.321161 26051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "df69f0ce60774a99874c1655d3049208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f9e7242bfb4433887d936becc85a7f"
I20250113 03:08:23.321215 26051 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:23.321166 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "df69f0ce60774a99874c1655d3049208" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cffa3d3db2478e954fcd9dcc84e093"
I20250113 03:08:23.321246 25900 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:23.321877 26051 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f0ce60774a99874c1655d3049208 in term 1.
I20250113 03:08:23.321990 25900 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df69f0ce60774a99874c1655d3049208 in term 1.
I20250113 03:08:23.322021 25941 leader_election.cc:304] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6f9e7242bfb4433887d936becc85a7f, df69f0ce60774a99874c1655d3049208; no voters:
I20250113 03:08:23.322080 26096 raft_consensus.cc:2798] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:23.322213 26096 raft_consensus.cc:695] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 LEADER]: Becoming Leader. State: Replica: df69f0ce60774a99874c1655d3049208, State: Running, Role: LEADER
I20250113 03:08:23.322285 26096 consensus_queue.cc:237] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:23.322770 25797 catalog_manager.cc:5526] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 reported cstate change: term changed from 0 to 1, leader changed from <none> to df69f0ce60774a99874c1655d3049208 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "df69f0ce60774a99874c1655d3049208" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:23.367172 26051 raft_consensus.cc:1270] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 1 FOLLOWER]: Refusing update from remote peer df69f0ce60774a99874c1655d3049208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:23.367209 25900 raft_consensus.cc:1270] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 1 FOLLOWER]: Refusing update from remote peer df69f0ce60774a99874c1655d3049208: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:23.367421 26102 consensus_queue.cc:1035] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:23.367504 26096 consensus_queue.cc:1035] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:23.368635 26109 mvcc.cc:204] Tried to move back new op lower bound from 7113677632990928896 to 7113677632808546304. Current Snapshot: MvccSnapshot[applied={T|T < 7113677632990928896}]
I20250113 03:08:33.901268 26138 raft_consensus.cc:988] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: : Instructing follower c0cffa3d3db2478e954fcd9dcc84e093 to start an election
I20250113 03:08:33.901350 26138 raft_consensus.cc:1076] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 LEADER]: Signalling peer c0cffa3d3db2478e954fcd9dcc84e093 to start an election
I20250113 03:08:33.901505 25900 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f"
dest_uuid: "c0cffa3d3db2478e954fcd9dcc84e093"
from {username='slave'} at 127.0.0.1:39360
I20250113 03:08:33.901592 25900 raft_consensus.cc:491] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:08:33.901614 25900 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:33.902230 25900 raft_consensus.cc:513] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:33.902383 25900 leader_election.cc:290] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [CANDIDATE]: Term 2 election: Requested vote from peers df69f0ce60774a99874c1655d3049208 (127.20.18.66:39215), d6f9e7242bfb4433887d936becc85a7f (127.20.18.67:38267)
I20250113 03:08:33.904798 25976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" candidate_term: 2 candidate_status { last_received { term: 1 index: 15218 } } ignore_live_leader: true dest_uuid: "df69f0ce60774a99874c1655d3049208"
I20250113 03:08:33.904800 26050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c39799ec9e43e7ae1e4a3e3c26480f" candidate_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" candidate_term: 2 candidate_status { last_received { term: 1 index: 15218 } } ignore_live_leader: true dest_uuid: "d6f9e7242bfb4433887d936becc85a7f"
I20250113 03:08:33.904872 25976 raft_consensus.cc:3049] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:08:33.904908 26050 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:33.904912 25976 raft_consensus.cc:738] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df69f0ce60774a99874c1655d3049208, State: Running, Role: LEADER
I20250113 03:08:33.904987 25976 consensus_queue.cc:260] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15218, Committed index: 15218, Last appended: 1.15218, Last appended by leader: 15218, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:33.905062 25976 raft_consensus.cc:3054] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:33.905540 26050 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0cffa3d3db2478e954fcd9dcc84e093 in term 2.
I20250113 03:08:33.905640 25976 raft_consensus.cc:2463] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0cffa3d3db2478e954fcd9dcc84e093 in term 2.
I20250113 03:08:33.905716 25865 leader_election.cc:304] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0cffa3d3db2478e954fcd9dcc84e093, d6f9e7242bfb4433887d936becc85a7f; no voters:
I20250113 03:08:33.905885 26156 raft_consensus.cc:2798] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:08:33.906009 26156 raft_consensus.cc:695] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 2 LEADER]: Becoming Leader. State: Replica: c0cffa3d3db2478e954fcd9dcc84e093, State: Running, Role: LEADER
I20250113 03:08:33.906064 26156 consensus_queue.cc:237] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15218, Committed index: 15218, Last appended: 1.15218, Last appended by leader: 15218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } }
I20250113 03:08:33.906445 25796 catalog_manager.cc:5526] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 reported cstate change: term changed from 1 to 2, leader changed from df69f0ce60774a99874c1655d3049208 (127.20.18.66) to c0cffa3d3db2478e954fcd9dcc84e093 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0cffa3d3db2478e954fcd9dcc84e093" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 36325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:33.915748 26050 raft_consensus.cc:1270] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 2 FOLLOWER]: Refusing update from remote peer c0cffa3d3db2478e954fcd9dcc84e093: Log matching property violated. Preceding OpId in replica: term: 1 index: 15218. Preceding OpId from leader: term: 2 index: 15220. (index mismatch)
I20250113 03:08:33.915793 25976 raft_consensus.cc:1270] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 2 FOLLOWER]: Refusing update from remote peer c0cffa3d3db2478e954fcd9dcc84e093: Log matching property violated. Preceding OpId in replica: term: 1 index: 15218. Preceding OpId from leader: term: 2 index: 15220. (index mismatch)
I20250113 03:08:33.915966 26156 consensus_queue.cc:1035] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f9e7242bfb4433887d936becc85a7f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15219, Last known committed idx: 15218, Time since last communication: 0.000s
I20250113 03:08:33.916066 26157 consensus_queue.cc:1035] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df69f0ce60774a99874c1655d3049208" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15219, Last known committed idx: 15218, Time since last communication: 0.000s
I20250113 03:08:33.920473 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:33.923053 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:33.923169 20553 tablet_replica.cc:331] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093: stopping tablet replica
I20250113 03:08:33.923228 20553 raft_consensus.cc:2238] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:08:33.923269 20553 raft_consensus.cc:2267] T b1c39799ec9e43e7ae1e4a3e3c26480f P c0cffa3d3db2478e954fcd9dcc84e093 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:33.936736 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:33.937489 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:33.940804 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:33.940901 20553 tablet_replica.cc:331] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208: stopping tablet replica
I20250113 03:08:33.940953 20553 raft_consensus.cc:2238] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:33.940999 20553 raft_consensus.cc:2267] T b1c39799ec9e43e7ae1e4a3e3c26480f P df69f0ce60774a99874c1655d3049208 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:33.961639 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:33.962438 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:33.965301 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:33.965417 20553 tablet_replica.cc:331] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f: stopping tablet replica
I20250113 03:08:33.965482 20553 raft_consensus.cc:2238] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:33.965515 20553 raft_consensus.cc:2267] T b1c39799ec9e43e7ae1e4a3e3c26480f P d6f9e7242bfb4433887d936becc85a7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:33.968360 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:33.969029 20553 master.cc:537] Master@127.20.18.126:43469 shutting down...
I20250113 03:08:33.971563 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:33.971614 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:33.971630 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 213b2706a6df443ba0e4707e5d7f4036: stopping tablet replica
I20250113 03:08:33.983330 20553 master.cc:559] Master@127.20.18.126:43469 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock (10937 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientCommitSequence
I20250113 03:08:34.003706 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:35157
I20250113 03:08:34.003851 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:34.005883 26181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:34.006453 26179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.007066 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:34.007129 26178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.007242 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:34.007275 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:34.007287 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737714007288 us; error 0 us; skew 500 ppm
I20250113 03:08:34.007819 20553 webserver.cc:458] Webserver started at http://127.20.18.126:42481/ using document root <none> and password file <none>
I20250113 03:08:34.007891 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:34.007936 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:34.007980 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:34.008237 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.009078 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:34.009477 26186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.009614 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.009653 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.009693 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:34.019263 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:34.019424 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:34.021620 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:35157
I20250113 03:08:34.032953 26239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:34.032977 26238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:35157 every 8 connection(s)
I20250113 03:08:34.033926 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab: Bootstrap starting.
I20250113 03:08:34.034185 26239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:34.034585 26239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab: No bootstrap required, opened a new log
I20250113 03:08:34.034685 26239 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER }
I20250113 03:08:34.034735 26239 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:34.034747 26239 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6a0768e7bb4ed6aecb19acd837d5ab, State: Initialized, Role: FOLLOWER
I20250113 03:08:34.034778 26239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER }
I20250113 03:08:34.034816 26239 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:34.034832 26239 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:34.034847 26239 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:34.035236 26239 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER }
I20250113 03:08:34.035281 26239 leader_election.cc:304] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6a0768e7bb4ed6aecb19acd837d5ab; no voters:
I20250113 03:08:34.035349 26239 leader_election.cc:290] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:34.035396 26242 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:34.035470 26239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:34.035522 26242 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 6f6a0768e7bb4ed6aecb19acd837d5ab, State: Running, Role: LEADER
I20250113 03:08:34.035576 26242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER }
I20250113 03:08:34.035749 26243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER } }
I20250113 03:08:34.035765 26244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6a0768e7bb4ed6aecb19acd837d5ab. Latest consensus state: current_term: 1 leader_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6a0768e7bb4ed6aecb19acd837d5ab" member_type: VOTER } }
I20250113 03:08:34.035813 26243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:34.035821 26244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:34.036087 26248 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:34.036222 26248 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:34.036720 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:34.036733 26248 catalog_manager.cc:1348] Generated new cluster ID: 2b40a8040738466ba24f6c3bf7877176
I20250113 03:08:34.036782 26248 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:34.044844 26248 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:34.045186 26248 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:34.047766 26248 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab: Generated new TSK 0
I20250113 03:08:34.047812 26248 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:34.052383 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:34.053635 26260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.055205 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:34.055718 26263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:34.055761 26261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.055891 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:34.055948 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:34.055959 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737714055960 us; error 0 us; skew 500 ppm
I20250113 03:08:34.056579 20553 webserver.cc:458] Webserver started at http://127.20.18.65:45397/ using document root <none> and password file <none>
I20250113 03:08:34.056668 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:34.056715 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:34.056761 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:34.057005 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "59d4736c25384f1b83a376bb4f4e3cd3"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.057770 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:34.058140 26268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.058233 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.058274 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "59d4736c25384f1b83a376bb4f4e3cd3"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.058307 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:34.065065 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:34.065250 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:34.065361 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:34.065519 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:34.065541 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.065569 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:34.065593 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.067906 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:37539
I20250113 03:08:34.067937 26331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:37539 every 8 connection(s)
I20250113 03:08:34.068244 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:34.069463 26336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:34.070159 26337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:34.070957 26339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.071379 26332 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35157
I20250113 03:08:34.071446 26332 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:34.071508 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:34.071555 26332 heartbeater.cc:510] Master 127.20.18.126:35157 requested a full tablet report, sending...
I20250113 03:08:34.071621 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:34.071653 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:34.071674 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737714071674 us; error 0 us; skew 500 ppm
I20250113 03:08:34.071799 26203 ts_manager.cc:194] Registered new tserver with Master: 59d4736c25384f1b83a376bb4f4e3cd3 (127.20.18.65:37539)
I20250113 03:08:34.072324 20553 webserver.cc:458] Webserver started at http://127.20.18.66:45891/ using document root <none> and password file <none>
I20250113 03:08:34.072403 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:34.072441 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:34.072482 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:34.072556 26203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42636
I20250113 03:08:34.072795 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "b7a540acea7145e892cdcd5592081a1b"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.073655 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:34.074031 26344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.074141 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.074180 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "b7a540acea7145e892cdcd5592081a1b"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.074219 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:34.082144 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:34.082281 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:34.082391 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:34.082545 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:34.082568 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.082595 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:34.082619 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.085412 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:42309
I20250113 03:08:34.085450 26407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:42309 every 8 connection(s)
I20250113 03:08:34.085739 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:34.086895 26411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:34.087594 26414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.088261 26408 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35157
I20250113 03:08:34.088320 26408 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:34.088347 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:34.088423 26408 heartbeater.cc:510] Master 127.20.18.126:35157 requested a full tablet report, sending...
I20250113 03:08:34.088621 26203 ts_manager.cc:194] Registered new tserver with Master: b7a540acea7145e892cdcd5592081a1b (127.20.18.66:42309)
W20250113 03:08:34.088858 26412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:34.088997 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:34.089030 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:34.089049 26203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42640
I20250113 03:08:34.089051 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737714089049 us; error 0 us; skew 500 ppm
I20250113 03:08:34.089671 20553 webserver.cc:458] Webserver started at http://127.20.18.67:39709/ using document root <none> and password file <none>
I20250113 03:08:34.089742 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:34.089787 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:34.089828 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:34.090076 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.090826 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:34.091197 26419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.091305 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.091342 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0"
format_stamp: "Formatted at 2025-01-13 03:08:34 on dist-test-slave-d4dl"
I20250113 03:08:34.091384 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:34.100059 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:34.100224 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:34.100348 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:34.100502 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:34.100550 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.100584 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:34.100610 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.103415 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:42791
I20250113 03:08:34.103468 26482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:42791 every 8 connection(s)
I20250113 03:08:34.105813 26483 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35157
I20250113 03:08:34.105855 26483 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:34.105933 26483 heartbeater.cc:510] Master 127.20.18.126:35157 requested a full tablet report, sending...
I20250113 03:08:34.106130 26203 ts_manager.cc:194] Registered new tserver with Master: 5900e19a1ca74f24b6e5aba5fa50c9d0 (127.20.18.67:42791)
I20250113 03:08:34.106415 26203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42646
I20250113 03:08:34.106818 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003251812s
I20250113 03:08:34.107100 20553 test_util.cc:274] Using random seed: -1031507237
I20250113 03:08:34.110983 26203 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42658:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:34.111097 26203 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:34.114524 26296 tablet_service.cc:1467] Processing CreateTablet for tablet 8bac9baf77514a27a864ec9ed6c637cf (DEFAULT_TABLE table=test-workload [id=649ed5b2aa654683b138f499a464f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:34.114632 26296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bac9baf77514a27a864ec9ed6c637cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:34.115615 26447 tablet_service.cc:1467] Processing CreateTablet for tablet 8bac9baf77514a27a864ec9ed6c637cf (DEFAULT_TABLE table=test-workload [id=649ed5b2aa654683b138f499a464f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:34.115669 26372 tablet_service.cc:1467] Processing CreateTablet for tablet 8bac9baf77514a27a864ec9ed6c637cf (DEFAULT_TABLE table=test-workload [id=649ed5b2aa654683b138f499a464f1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:34.115697 26447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bac9baf77514a27a864ec9ed6c637cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:34.115775 26372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bac9baf77514a27a864ec9ed6c637cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:34.115953 26498 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: Bootstrap starting.
I20250113 03:08:34.116394 26498 tablet_bootstrap.cc:654] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:34.116984 26498 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: No bootstrap required, opened a new log
I20250113 03:08:34.117014 26500 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: Bootstrap starting.
I20250113 03:08:34.117033 26498 ts_tablet_manager.cc:1397] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:34.117233 26498 raft_consensus.cc:357] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.117277 26498 raft_consensus.cc:383] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:34.117298 26498 raft_consensus.cc:738] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d4736c25384f1b83a376bb4f4e3cd3, State: Initialized, Role: FOLLOWER
I20250113 03:08:34.117360 26498 consensus_queue.cc:260] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.117449 26500 tablet_bootstrap.cc:654] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:34.117502 26498 ts_tablet_manager.cc:1428] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.117594 26332 heartbeater.cc:502] Master 127.20.18.126:35157 was elected leader, sending a full tablet report...
I20250113 03:08:34.117802 26501 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: Bootstrap starting.
I20250113 03:08:34.118075 26500 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: No bootstrap required, opened a new log
I20250113 03:08:34.118115 26500 ts_tablet_manager.cc:1397] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:34.118129 26501 tablet_bootstrap.cc:654] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:34.118247 26500 raft_consensus.cc:357] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.118286 26500 raft_consensus.cc:383] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:34.118307 26500 raft_consensus.cc:738] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5900e19a1ca74f24b6e5aba5fa50c9d0, State: Initialized, Role: FOLLOWER
I20250113 03:08:34.118355 26500 consensus_queue.cc:260] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.118505 26500 ts_tablet_manager.cc:1428] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.118525 26483 heartbeater.cc:502] Master 127.20.18.126:35157 was elected leader, sending a full tablet report...
I20250113 03:08:34.118609 26501 tablet_bootstrap.cc:492] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: No bootstrap required, opened a new log
I20250113 03:08:34.118675 26501 ts_tablet_manager.cc:1397] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:34.118798 26501 raft_consensus.cc:357] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.118852 26501 raft_consensus.cc:383] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:34.118873 26501 raft_consensus.cc:738] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a540acea7145e892cdcd5592081a1b, State: Initialized, Role: FOLLOWER
I20250113 03:08:34.118921 26501 consensus_queue.cc:260] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.119050 26501 ts_tablet_manager.cc:1428] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:34.119071 26408 heartbeater.cc:502] Master 127.20.18.126:35157 was elected leader, sending a full tablet report...
I20250113 03:08:34.137491 26505 raft_consensus.cc:491] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:34.137557 26505 raft_consensus.cc:513] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.137691 26505 leader_election.cc:290] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a540acea7145e892cdcd5592081a1b (127.20.18.66:42309), 59d4736c25384f1b83a376bb4f4e3cd3 (127.20.18.65:37539)
I20250113 03:08:34.139750 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a540acea7145e892cdcd5592081a1b" is_pre_election: true
I20250113 03:08:34.139830 26382 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5900e19a1ca74f24b6e5aba5fa50c9d0 in term 0.
I20250113 03:08:34.139866 26306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" is_pre_election: true
I20250113 03:08:34.139942 26306 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5900e19a1ca74f24b6e5aba5fa50c9d0 in term 0.
I20250113 03:08:34.139968 26421 leader_election.cc:304] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5900e19a1ca74f24b6e5aba5fa50c9d0, b7a540acea7145e892cdcd5592081a1b; no voters:
I20250113 03:08:34.140025 26505 raft_consensus.cc:2798] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:34.140055 26505 raft_consensus.cc:491] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:34.140082 26505 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:34.140661 26505 raft_consensus.cc:513] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.140770 26505 leader_election.cc:290] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [CANDIDATE]: Term 1 election: Requested vote from peers b7a540acea7145e892cdcd5592081a1b (127.20.18.66:42309), 59d4736c25384f1b83a376bb4f4e3cd3 (127.20.18.65:37539)
I20250113 03:08:34.140880 26382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a540acea7145e892cdcd5592081a1b"
I20250113 03:08:34.140938 26382 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:34.140918 26306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d4736c25384f1b83a376bb4f4e3cd3"
I20250113 03:08:34.141003 26306 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:34.141494 26382 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5900e19a1ca74f24b6e5aba5fa50c9d0 in term 1.
I20250113 03:08:34.141611 26306 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5900e19a1ca74f24b6e5aba5fa50c9d0 in term 1.
I20250113 03:08:34.141649 26421 leader_election.cc:304] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5900e19a1ca74f24b6e5aba5fa50c9d0, b7a540acea7145e892cdcd5592081a1b; no voters:
I20250113 03:08:34.141707 26505 raft_consensus.cc:2798] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:34.141820 26505 raft_consensus.cc:695] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5900e19a1ca74f24b6e5aba5fa50c9d0, State: Running, Role: LEADER
I20250113 03:08:34.141868 26505 consensus_queue.cc:237] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:34.142380 26203 catalog_manager.cc:5526] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5900e19a1ca74f24b6e5aba5fa50c9d0 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:34.158514 26382 raft_consensus.cc:1270] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 1 FOLLOWER]: Refusing update from remote peer 5900e19a1ca74f24b6e5aba5fa50c9d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:34.158541 26306 raft_consensus.cc:1270] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 1 FOLLOWER]: Refusing update from remote peer 5900e19a1ca74f24b6e5aba5fa50c9d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:34.158756 26508 consensus_queue.cc:1035] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:34.158883 26508 consensus_queue.cc:1035] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:44.768442 26535 raft_consensus.cc:988] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: : Instructing follower 59d4736c25384f1b83a376bb4f4e3cd3 to start an election
I20250113 03:08:44.768520 26535 raft_consensus.cc:1076] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 LEADER]: Signalling peer 59d4736c25384f1b83a376bb4f4e3cd3 to start an election
I20250113 03:08:44.768728 26305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf"
dest_uuid: "59d4736c25384f1b83a376bb4f4e3cd3"
from {username='slave'} at 127.0.0.1:35122
I20250113 03:08:44.768790 26305 raft_consensus.cc:491] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:08:44.768815 26305 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:44.769472 26305 raft_consensus.cc:513] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:44.769647 26305 leader_election.cc:290] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [CANDIDATE]: Term 2 election: Requested vote from peers b7a540acea7145e892cdcd5592081a1b (127.20.18.66:42309), 5900e19a1ca74f24b6e5aba5fa50c9d0 (127.20.18.67:42791)
I20250113 03:08:44.772167 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 15781 } } ignore_live_leader: true dest_uuid: "b7a540acea7145e892cdcd5592081a1b"
I20250113 03:08:44.772181 26457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bac9baf77514a27a864ec9ed6c637cf" candidate_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 15781 } } ignore_live_leader: true dest_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0"
I20250113 03:08:44.772240 26381 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:44.772255 26457 raft_consensus.cc:3049] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:08:44.772272 26457 raft_consensus.cc:738] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5900e19a1ca74f24b6e5aba5fa50c9d0, State: Running, Role: LEADER
I20250113 03:08:44.772330 26457 consensus_queue.cc:260] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15781, Committed index: 15781, Last appended: 1.15781, Last appended by leader: 15781, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:44.772411 26457 raft_consensus.cc:3054] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:44.772923 26381 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d4736c25384f1b83a376bb4f4e3cd3 in term 2.
I20250113 03:08:44.773025 26457 raft_consensus.cc:2463] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d4736c25384f1b83a376bb4f4e3cd3 in term 2.
I20250113 03:08:44.773133 26270 leader_election.cc:304] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d4736c25384f1b83a376bb4f4e3cd3, b7a540acea7145e892cdcd5592081a1b; no voters:
I20250113 03:08:44.773337 26560 raft_consensus.cc:2798] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:08:44.773447 26560 raft_consensus.cc:695] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 2 LEADER]: Becoming Leader. State: Replica: 59d4736c25384f1b83a376bb4f4e3cd3, State: Running, Role: LEADER
I20250113 03:08:44.773504 26560 consensus_queue.cc:237] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15781, Committed index: 15781, Last appended: 1.15781, Last appended by leader: 15781, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } }
I20250113 03:08:44.773918 26200 catalog_manager.cc:5526] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 reported cstate change: term changed from 1 to 2, leader changed from 5900e19a1ca74f24b6e5aba5fa50c9d0 (127.20.18.67) to 59d4736c25384f1b83a376bb4f4e3cd3 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d4736c25384f1b83a376bb4f4e3cd3" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37539 } health_report { overall_health: HEALTHY } } }
I20250113 03:08:44.787371 26457 raft_consensus.cc:1270] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 2 FOLLOWER]: Refusing update from remote peer 59d4736c25384f1b83a376bb4f4e3cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 15781. Preceding OpId from leader: term: 2 index: 15783. (index mismatch)
I20250113 03:08:44.787377 26381 raft_consensus.cc:1270] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 2 FOLLOWER]: Refusing update from remote peer 59d4736c25384f1b83a376bb4f4e3cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 15781. Preceding OpId from leader: term: 2 index: 15783. (index mismatch)
I20250113 03:08:44.787592 26561 consensus_queue.cc:1035] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5900e19a1ca74f24b6e5aba5fa50c9d0" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15782, Last known committed idx: 15781, Time since last communication: 0.000s
I20250113 03:08:44.787698 26560 consensus_queue.cc:1035] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a540acea7145e892cdcd5592081a1b" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15782, Last known committed idx: 15781, Time since last communication: 0.000s
W20250113 03:08:44.790252 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Cannot finalize transaction in state: OPEN
W20250113 03:08:44.790604 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction 0 already open
W20250113 03:08:44.792902 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:08:44.793210 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction 0 commit already in progress
W20250113 03:08:44.794489 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:08:44.794756 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:08:44.795022 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction 0 commit already in progress
W20250113 03:08:44.795267 26574 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32826: Illegal state: Transaction metadata for transaction 0 already set
I20250113 03:08:44.795564 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:44.797763 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:44.797849 20553 tablet_replica.cc:331] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3: stopping tablet replica
I20250113 03:08:44.797891 20553 raft_consensus.cc:2238] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:08:44.797960 20553 raft_consensus.cc:2267] T 8bac9baf77514a27a864ec9ed6c637cf P 59d4736c25384f1b83a376bb4f4e3cd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:44.815935 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:44.816686 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:44.819438 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:44.819541 20553 tablet_replica.cc:331] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b: stopping tablet replica
I20250113 03:08:44.819587 20553 raft_consensus.cc:2238] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:44.819628 20553 raft_consensus.cc:2267] T 8bac9baf77514a27a864ec9ed6c637cf P b7a540acea7145e892cdcd5592081a1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:44.833024 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:44.833751 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:44.836380 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:44.836500 20553 tablet_replica.cc:331] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0: stopping tablet replica
I20250113 03:08:44.836581 20553 raft_consensus.cc:2238] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:44.836612 20553 raft_consensus.cc:2267] T 8bac9baf77514a27a864ec9ed6c637cf P 5900e19a1ca74f24b6e5aba5fa50c9d0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:44.848398 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:44.849081 20553 master.cc:537] Master@127.20.18.126:35157 shutting down...
I20250113 03:08:44.850744 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:44.850816 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:44.850844 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6f6a0768e7bb4ed6aecb19acd837d5ab: stopping tablet replica
I20250113 03:08:44.862356 20553 master.cc:559] Master@127.20.18.126:35157 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientCommitSequence (10877 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientAbortSequence
I20250113 03:08:44.881609 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:42673
I20250113 03:08:44.881749 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:44.883597 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.883862 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.884557 26585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:44.885267 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:44.885375 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:44.885402 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:44.885414 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737724885415 us; error 0 us; skew 500 ppm
I20250113 03:08:44.886025 20553 webserver.cc:458] Webserver started at http://127.20.18.126:41973/ using document root <none> and password file <none>
I20250113 03:08:44.886099 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:44.886142 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:44.886191 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:44.886466 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "a3760cfdb55844e983e4dcbc76f82cf5"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.887302 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250113 03:08:44.887683 26590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.887811 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.887856 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "a3760cfdb55844e983e4dcbc76f82cf5"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.887889 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:44.905788 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:44.905939 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:44.908169 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:42673
I20250113 03:08:44.911355 26642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:42673 every 8 connection(s)
I20250113 03:08:44.911468 26643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:44.913533 26643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5: Bootstrap starting.
I20250113 03:08:44.913856 26643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:44.914268 26643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5: No bootstrap required, opened a new log
I20250113 03:08:44.914394 26643 raft_consensus.cc:357] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER }
I20250113 03:08:44.914439 26643 raft_consensus.cc:383] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:44.914453 26643 raft_consensus.cc:738] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3760cfdb55844e983e4dcbc76f82cf5, State: Initialized, Role: FOLLOWER
I20250113 03:08:44.914500 26643 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER }
I20250113 03:08:44.914537 26643 raft_consensus.cc:397] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:44.914566 26643 raft_consensus.cc:491] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:44.914593 26643 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:44.915038 26643 raft_consensus.cc:513] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER }
I20250113 03:08:44.915089 26643 leader_election.cc:304] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3760cfdb55844e983e4dcbc76f82cf5; no voters:
I20250113 03:08:44.915187 26643 leader_election.cc:290] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:44.915233 26646 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:44.915311 26643 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:44.915393 26646 raft_consensus.cc:695] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 1 LEADER]: Becoming Leader. State: Replica: a3760cfdb55844e983e4dcbc76f82cf5, State: Running, Role: LEADER
I20250113 03:08:44.915450 26646 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER }
I20250113 03:08:44.915638 26647 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER } }
I20250113 03:08:44.915696 26647 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:44.916016 26648 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3760cfdb55844e983e4dcbc76f82cf5. Latest consensus state: current_term: 1 leader_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3760cfdb55844e983e4dcbc76f82cf5" member_type: VOTER } }
I20250113 03:08:44.916028 26651 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:44.916113 26648 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:44.916219 26651 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:44.916572 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:44.916705 26651 catalog_manager.cc:1348] Generated new cluster ID: d25cf7f3ae3344c289d398200180cd79
I20250113 03:08:44.916757 26651 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:44.930725 26651 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:44.931140 26651 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:44.936084 26651 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5: Generated new TSK 0
I20250113 03:08:44.936147 26651 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:44.948236 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:44.949687 26665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.950486 26666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.951076 26668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:44.951779 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:44.951844 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:44.951877 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:44.951890 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737724951890 us; error 0 us; skew 500 ppm
I20250113 03:08:44.952509 20553 webserver.cc:458] Webserver started at http://127.20.18.65:38155/ using document root <none> and password file <none>
I20250113 03:08:44.952615 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:44.952656 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:44.952705 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:44.952976 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "e35a5ad42068440e95d647fcba6b452c"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.953783 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:44.954200 26673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.954327 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.954373 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "e35a5ad42068440e95d647fcba6b452c"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.954407 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:44.958765 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:44.958909 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:44.959010 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:44.959146 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:44.959167 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.959189 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:44.959205 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.961709 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:39797
I20250113 03:08:44.961831 26736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:39797 every 8 connection(s)
I20250113 03:08:44.962088 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:44.963256 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.963819 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.964426 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:44.965176 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:44.965289 26737 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42673
I20250113 03:08:44.965297 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:08:44.965360 26737 heartbeater.cc:463] Registering TS with master...
W20250113 03:08:44.965369 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:44.965390 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737724965390 us; error 0 us; skew 500 ppm
I20250113 03:08:44.965456 26737 heartbeater.cc:510] Master 127.20.18.126:42673 requested a full tablet report, sending...
I20250113 03:08:44.965690 26607 ts_manager.cc:194] Registered new tserver with Master: e35a5ad42068440e95d647fcba6b452c (127.20.18.65:39797)
I20250113 03:08:44.966125 20553 webserver.cc:458] Webserver started at http://127.20.18.66:33751/ using document root <none> and password file <none>
I20250113 03:08:44.966200 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:44.966250 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:44.966298 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:44.966357 26607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33120
I20250113 03:08:44.966564 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "eddbf67ce66b418096a4a696df817fb7"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.967384 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:44.967757 26749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.967873 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.967911 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "eddbf67ce66b418096a4a696df817fb7"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.967952 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:44.973032 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:44.973173 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:44.973294 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:44.973445 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:44.973470 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.973497 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:44.973521 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.976089 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:37083
I20250113 03:08:44.976141 26812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:37083 every 8 connection(s)
I20250113 03:08:44.976416 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:44.977893 26817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:44.978147 26816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:44.978852 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:44.979133 26813 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42673
I20250113 03:08:44.979187 26813 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:44.979280 26813 heartbeater.cc:510] Master 127.20.18.126:42673 requested a full tablet report, sending...
I20250113 03:08:44.979466 26607 ts_manager.cc:194] Registered new tserver with Master: eddbf67ce66b418096a4a696df817fb7 (127.20.18.66:37083)
W20250113 03:08:44.979534 26819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:44.979705 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:44.979737 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:44.979753 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737724979753 us; error 0 us; skew 500 ppm
I20250113 03:08:44.979907 26607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33136
I20250113 03:08:44.980352 20553 webserver.cc:458] Webserver started at http://127.20.18.67:35883/ using document root <none> and password file <none>
I20250113 03:08:44.980412 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:44.980443 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:44.980473 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:44.980721 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "d9f7665aa3704f9fac737c1f8abee9b2"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.981514 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:44.981904 26824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.982015 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:44.982059 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "d9f7665aa3704f9fac737c1f8abee9b2"
format_stamp: "Formatted at 2025-01-13 03:08:44 on dist-test-slave-d4dl"
I20250113 03:08:44.982090 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:45.000257 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:45.000433 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:45.000576 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:45.000736 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:45.000758 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:45.000788 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:45.000799 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:45.003571 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:38653
I20250113 03:08:45.003629 26887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:38653 every 8 connection(s)
I20250113 03:08:45.006352 26888 heartbeater.cc:346] Connected to a master server at 127.20.18.126:42673
I20250113 03:08:45.006402 26888 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:45.006500 26888 heartbeater.cc:510] Master 127.20.18.126:42673 requested a full tablet report, sending...
I20250113 03:08:45.006721 26607 ts_manager.cc:194] Registered new tserver with Master: d9f7665aa3704f9fac737c1f8abee9b2 (127.20.18.67:38653)
I20250113 03:08:45.007047 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003287265s
I20250113 03:08:45.007055 26607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33144
I20250113 03:08:45.007381 20553 test_util.cc:274] Using random seed: -1020606955
I20250113 03:08:45.011371 26607 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:45.011497 26607 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:45.015745 26701 tablet_service.cc:1467] Processing CreateTablet for tablet 240bb50b57724ec882435bb3939a8482 (DEFAULT_TABLE table=test-workload [id=b4bd853637044e7894bcf8739ff7e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:45.015892 26701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240bb50b57724ec882435bb3939a8482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:45.015973 26852 tablet_service.cc:1467] Processing CreateTablet for tablet 240bb50b57724ec882435bb3939a8482 (DEFAULT_TABLE table=test-workload [id=b4bd853637044e7894bcf8739ff7e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:45.016049 26852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240bb50b57724ec882435bb3939a8482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:45.017298 26777 tablet_service.cc:1467] Processing CreateTablet for tablet 240bb50b57724ec882435bb3939a8482 (DEFAULT_TABLE table=test-workload [id=b4bd853637044e7894bcf8739ff7e5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:45.017439 26777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240bb50b57724ec882435bb3939a8482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:45.017264 26903 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: Bootstrap starting.
I20250113 03:08:45.018236 26905 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: Bootstrap starting.
I20250113 03:08:45.018569 26905 tablet_bootstrap.cc:654] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:45.018590 26903 tablet_bootstrap.cc:654] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:45.018908 26904 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: Bootstrap starting.
I20250113 03:08:45.019160 26903 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: No bootstrap required, opened a new log
I20250113 03:08:45.019199 26903 ts_tablet_manager.cc:1397] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:08:45.019304 26904 tablet_bootstrap.cc:654] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:45.019313 26903 raft_consensus.cc:357] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.019395 26903 raft_consensus.cc:383] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:45.019438 26903 raft_consensus.cc:738] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35a5ad42068440e95d647fcba6b452c, State: Initialized, Role: FOLLOWER
I20250113 03:08:45.019512 26903 consensus_queue.cc:260] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.019671 26903 ts_tablet_manager.cc:1428] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:45.019691 26737 heartbeater.cc:502] Master 127.20.18.126:42673 was elected leader, sending a full tablet report...
I20250113 03:08:45.019843 26905 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: No bootstrap required, opened a new log
I20250113 03:08:45.019881 26905 ts_tablet_manager.cc:1397] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:08:45.019889 26904 tablet_bootstrap.cc:492] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: No bootstrap required, opened a new log
I20250113 03:08:45.019945 26904 ts_tablet_manager.cc:1397] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:45.020015 26905 raft_consensus.cc:357] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.020061 26905 raft_consensus.cc:383] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:45.020076 26904 raft_consensus.cc:357] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.020149 26904 raft_consensus.cc:383] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:45.020080 26905 raft_consensus.cc:738] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddbf67ce66b418096a4a696df817fb7, State: Initialized, Role: FOLLOWER
I20250113 03:08:45.020180 26904 raft_consensus.cc:738] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f7665aa3704f9fac737c1f8abee9b2, State: Initialized, Role: FOLLOWER
I20250113 03:08:45.020247 26904 consensus_queue.cc:260] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.020401 26904 ts_tablet_manager.cc:1428] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:45.020462 26888 heartbeater.cc:502] Master 127.20.18.126:42673 was elected leader, sending a full tablet report...
I20250113 03:08:45.020510 26905 consensus_queue.cc:260] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.020692 26905 ts_tablet_manager.cc:1428] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:45.020714 26813 heartbeater.cc:502] Master 127.20.18.126:42673 was elected leader, sending a full tablet report...
I20250113 03:08:45.028651 26911 raft_consensus.cc:491] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:45.028740 26911 raft_consensus.cc:513] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.028905 26911 leader_election.cc:290] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35a5ad42068440e95d647fcba6b452c (127.20.18.65:39797), d9f7665aa3704f9fac737c1f8abee9b2 (127.20.18.67:38653)
I20250113 03:08:45.031916 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "eddbf67ce66b418096a4a696df817fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" is_pre_election: true
I20250113 03:08:45.031912 26711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "eddbf67ce66b418096a4a696df817fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35a5ad42068440e95d647fcba6b452c" is_pre_election: true
I20250113 03:08:45.032030 26862 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eddbf67ce66b418096a4a696df817fb7 in term 0.
I20250113 03:08:45.032035 26711 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eddbf67ce66b418096a4a696df817fb7 in term 0.
I20250113 03:08:45.032156 26751 leader_election.cc:304] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e35a5ad42068440e95d647fcba6b452c, eddbf67ce66b418096a4a696df817fb7; no voters:
I20250113 03:08:45.032263 26911 raft_consensus.cc:2798] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:45.032297 26911 raft_consensus.cc:491] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:45.032315 26911 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:45.032891 26911 raft_consensus.cc:513] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.033035 26911 leader_election.cc:290] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [CANDIDATE]: Term 1 election: Requested vote from peers e35a5ad42068440e95d647fcba6b452c (127.20.18.65:39797), d9f7665aa3704f9fac737c1f8abee9b2 (127.20.18.67:38653)
I20250113 03:08:45.033188 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "eddbf67ce66b418096a4a696df817fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9f7665aa3704f9fac737c1f8abee9b2"
I20250113 03:08:45.033202 26711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "eddbf67ce66b418096a4a696df817fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35a5ad42068440e95d647fcba6b452c"
I20250113 03:08:45.033262 26862 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:45.033263 26711 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:45.033833 26862 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eddbf67ce66b418096a4a696df817fb7 in term 1.
I20250113 03:08:45.033872 26711 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eddbf67ce66b418096a4a696df817fb7 in term 1.
I20250113 03:08:45.033986 26752 leader_election.cc:304] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9f7665aa3704f9fac737c1f8abee9b2, eddbf67ce66b418096a4a696df817fb7; no voters:
I20250113 03:08:45.034072 26911 raft_consensus.cc:2798] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:45.034178 26911 raft_consensus.cc:695] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 LEADER]: Becoming Leader. State: Replica: eddbf67ce66b418096a4a696df817fb7, State: Running, Role: LEADER
I20250113 03:08:45.034219 26911 consensus_queue.cc:237] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:45.034667 26607 catalog_manager.cc:5526] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eddbf67ce66b418096a4a696df817fb7 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "eddbf67ce66b418096a4a696df817fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:45.050218 26862 raft_consensus.cc:1270] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 1 FOLLOWER]: Refusing update from remote peer eddbf67ce66b418096a4a696df817fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:45.050268 26711 raft_consensus.cc:1270] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 1 FOLLOWER]: Refusing update from remote peer eddbf67ce66b418096a4a696df817fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:45.050427 26913 consensus_queue.cc:1035] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:45.050525 26911 consensus_queue.cc:1035] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:45.053102 26920 mvcc.cc:204] Tried to move back new op lower bound from 7113677721805524992 to 7113677721740636160. Current Snapshot: MvccSnapshot[applied={T|T < 7113677721805152256 or (T in {7113677721805524992})}]
I20250113 03:08:55.549677 26953 raft_consensus.cc:988] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: : Instructing follower e35a5ad42068440e95d647fcba6b452c to start an election
I20250113 03:08:55.549752 26953 raft_consensus.cc:1076] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 LEADER]: Signalling peer e35a5ad42068440e95d647fcba6b452c to start an election
I20250113 03:08:55.549942 26710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "240bb50b57724ec882435bb3939a8482"
dest_uuid: "e35a5ad42068440e95d647fcba6b452c"
from {username='slave'} at 127.0.0.1:43834
I20250113 03:08:55.550009 26710 raft_consensus.cc:491] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:08:55.550029 26710 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:55.550583 26710 raft_consensus.cc:513] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:55.550724 26710 leader_election.cc:290] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [CANDIDATE]: Term 2 election: Requested vote from peers eddbf67ce66b418096a4a696df817fb7 (127.20.18.66:37083), d9f7665aa3704f9fac737c1f8abee9b2 (127.20.18.67:38653)
I20250113 03:08:55.553249 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "e35a5ad42068440e95d647fcba6b452c" candidate_term: 2 candidate_status { last_received { term: 1 index: 15714 } } ignore_live_leader: true dest_uuid: "d9f7665aa3704f9fac737c1f8abee9b2"
I20250113 03:08:55.553259 26787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240bb50b57724ec882435bb3939a8482" candidate_uuid: "e35a5ad42068440e95d647fcba6b452c" candidate_term: 2 candidate_status { last_received { term: 1 index: 15714 } } ignore_live_leader: true dest_uuid: "eddbf67ce66b418096a4a696df817fb7"
I20250113 03:08:55.553335 26862 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:55.553368 26787 raft_consensus.cc:3049] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:08:55.553402 26787 raft_consensus.cc:738] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eddbf67ce66b418096a4a696df817fb7, State: Running, Role: LEADER
I20250113 03:08:55.553458 26787 consensus_queue.cc:260] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15714, Committed index: 15714, Last appended: 1.15714, Last appended by leader: 15714, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:55.553539 26787 raft_consensus.cc:3054] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:08:55.553897 26862 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35a5ad42068440e95d647fcba6b452c in term 2.
I20250113 03:08:55.554090 26676 leader_election.cc:304] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9f7665aa3704f9fac737c1f8abee9b2, e35a5ad42068440e95d647fcba6b452c; no voters:
I20250113 03:08:55.554100 26787 raft_consensus.cc:2463] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35a5ad42068440e95d647fcba6b452c in term 2.
I20250113 03:08:55.554280 26969 raft_consensus.cc:2798] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:08:55.554358 26969 raft_consensus.cc:695] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 2 LEADER]: Becoming Leader. State: Replica: e35a5ad42068440e95d647fcba6b452c, State: Running, Role: LEADER
I20250113 03:08:55.554440 26969 consensus_queue.cc:237] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15714, Committed index: 15714, Last appended: 1.15714, Last appended by leader: 15714, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } }
I20250113 03:08:55.554837 26606 catalog_manager.cc:5526] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c reported cstate change: term changed from 1 to 2, leader changed from eddbf67ce66b418096a4a696df817fb7 (127.20.18.66) to e35a5ad42068440e95d647fcba6b452c (127.20.18.65). New cstate: current_term: 2 leader_uuid: "e35a5ad42068440e95d647fcba6b452c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35a5ad42068440e95d647fcba6b452c" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 } health_report { overall_health: UNKNOWN } } }
I20250113 03:08:55.562399 26862 raft_consensus.cc:1270] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 2 FOLLOWER]: Refusing update from remote peer e35a5ad42068440e95d647fcba6b452c: Log matching property violated. Preceding OpId in replica: term: 1 index: 15714. Preceding OpId from leader: term: 2 index: 15716. (index mismatch)
I20250113 03:08:55.562405 26787 raft_consensus.cc:1270] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 2 FOLLOWER]: Refusing update from remote peer e35a5ad42068440e95d647fcba6b452c: Log matching property violated. Preceding OpId in replica: term: 1 index: 15714. Preceding OpId from leader: term: 2 index: 15716. (index mismatch)
I20250113 03:08:55.562563 26970 consensus_queue.cc:1035] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f7665aa3704f9fac737c1f8abee9b2" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15715, Last known committed idx: 15714, Time since last communication: 0.000s
I20250113 03:08:55.562634 26969 consensus_queue.cc:1035] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eddbf67ce66b418096a4a696df817fb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15715, Last known committed idx: 15714, Time since last communication: 0.000s
W20250113 03:08:55.569604 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:08:55.569950 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 1 already exists
W20250113 03:08:55.570232 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20250113 03:08:55.570490 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20250113 03:08:55.570767 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 0 already exists
W20250113 03:08:55.571055 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 1 already exists
W20250113 03:08:55.571336 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 0 already set
W20250113 03:08:55.571619 26983 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56124: Illegal state: Transaction metadata for transaction 1 already set
I20250113 03:08:55.571905 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:08:55.574379 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:55.574470 20553 tablet_replica.cc:331] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c: stopping tablet replica
I20250113 03:08:55.574505 20553 raft_consensus.cc:2238] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:08:55.574544 20553 raft_consensus.cc:2267] T 240bb50b57724ec882435bb3939a8482 P e35a5ad42068440e95d647fcba6b452c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:55.580489 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:08:55.581194 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:08:55.583989 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:55.584077 20553 tablet_replica.cc:331] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7: stopping tablet replica
I20250113 03:08:55.584120 20553 raft_consensus.cc:2238] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:55.584146 20553 raft_consensus.cc:2267] T 240bb50b57724ec882435bb3939a8482 P eddbf67ce66b418096a4a696df817fb7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:55.587450 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:08:55.588088 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:08:55.591254 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:08:55.591357 20553 tablet_replica.cc:331] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2: stopping tablet replica
I20250113 03:08:55.591398 20553 raft_consensus.cc:2238] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:08:55.591434 20553 raft_consensus.cc:2267] T 240bb50b57724ec882435bb3939a8482 P d9f7665aa3704f9fac737c1f8abee9b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:55.613241 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:08:55.613894 20553 master.cc:537] Master@127.20.18.126:42673 shutting down...
I20250113 03:08:55.616438 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:08:55.616492 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:08:55.616508 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P a3760cfdb55844e983e4dcbc76f82cf5: stopping tablet replica
I20250113 03:08:55.628198 20553 master.cc:559] Master@127.20.18.126:42673 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientAbortSequence (10765 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun
I20250113 03:08:55.646840 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:35427
I20250113 03:08:55.646981 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:55.648761 26991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.649597 20553 server_base.cc:1034] running on GCE node
W20250113 03:08:55.649797 26994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.650435 26992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.650556 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:55.650585 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:55.650596 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737735650597 us; error 0 us; skew 500 ppm
I20250113 03:08:55.651113 20553 webserver.cc:458] Webserver started at http://127.20.18.126:42465/ using document root <none> and password file <none>
I20250113 03:08:55.651183 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:55.651242 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:55.651288 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:55.651559 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "90bb1fc13245418d84477704f1c9db51"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.652352 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:55.652793 26999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.652917 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.652956 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "90bb1fc13245418d84477704f1c9db51"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.652993 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:55.662957 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:55.663092 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:55.665263 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:35427
I20250113 03:08:55.671355 27051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:35427 every 8 connection(s)
I20250113 03:08:55.672765 27052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:55.673631 27052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51: Bootstrap starting.
I20250113 03:08:55.673928 27052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:55.674319 27052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51: No bootstrap required, opened a new log
I20250113 03:08:55.674422 27052 raft_consensus.cc:357] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER }
I20250113 03:08:55.674464 27052 raft_consensus.cc:383] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:55.674477 27052 raft_consensus.cc:738] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bb1fc13245418d84477704f1c9db51, State: Initialized, Role: FOLLOWER
I20250113 03:08:55.674521 27052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER }
I20250113 03:08:55.674556 27052 raft_consensus.cc:397] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:08:55.674584 27052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:08:55.674608 27052 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:55.675037 27052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER }
I20250113 03:08:55.675083 27052 leader_election.cc:304] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90bb1fc13245418d84477704f1c9db51; no voters:
I20250113 03:08:55.675175 27052 leader_election.cc:290] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:08:55.675220 27055 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:55.675299 27052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:08:55.675360 27055 raft_consensus.cc:695] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 1 LEADER]: Becoming Leader. State: Replica: 90bb1fc13245418d84477704f1c9db51, State: Running, Role: LEADER
I20250113 03:08:55.675406 27055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER }
I20250113 03:08:55.675573 27056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90bb1fc13245418d84477704f1c9db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER } }
I20250113 03:08:55.675580 27057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bb1fc13245418d84477704f1c9db51. Latest consensus state: current_term: 1 leader_uuid: "90bb1fc13245418d84477704f1c9db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb1fc13245418d84477704f1c9db51" member_type: VOTER } }
I20250113 03:08:55.675655 27056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:55.675668 27057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [sys.catalog]: This master's current role is: LEADER
I20250113 03:08:55.675913 27060 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:08:55.676029 27060 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:08:55.676489 27060 catalog_manager.cc:1348] Generated new cluster ID: 7d15d27b8bfd4c32af887b45e33cc429
I20250113 03:08:55.676501 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:08:55.676575 27060 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:08:55.695245 27060 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:08:55.695626 27060 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:08:55.700057 27060 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51: Generated new TSK 0
I20250113 03:08:55.700114 27060 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:08:55.708484 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:55.709761 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.710417 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.711159 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.711872 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:55.711962 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:55.711988 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:55.712000 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737735712000 us; error 0 us; skew 500 ppm
I20250113 03:08:55.712596 20553 webserver.cc:458] Webserver started at http://127.20.18.65:45141/ using document root <none> and password file <none>
I20250113 03:08:55.712667 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:55.712698 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:55.712729 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:55.712975 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "85866f65084942ffb481710be83cd896"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.713703 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:55.714068 27081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.714179 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.714216 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "85866f65084942ffb481710be83cd896"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.714257 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:55.732933 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:55.733103 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:55.733212 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:55.733374 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:55.733397 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.733425 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:55.733448 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.735809 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:35135
I20250113 03:08:55.736027 27144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:35135 every 8 connection(s)
I20250113 03:08:55.736189 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:55.737381 27149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.738042 27152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.738692 27150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.739104 27145 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35427
I20250113 03:08:55.739151 27145 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:55.739233 27145 heartbeater.cc:510] Master 127.20.18.126:35427 requested a full tablet report, sending...
I20250113 03:08:55.739439 27016 ts_manager.cc:194] Registered new tserver with Master: 85866f65084942ffb481710be83cd896 (127.20.18.65:35135)
I20250113 03:08:55.739660 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:55.739775 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:08:55.739796 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:55.739810 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737735739811 us; error 0 us; skew 500 ppm
I20250113 03:08:55.739992 27016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40790
I20250113 03:08:55.740522 20553 webserver.cc:458] Webserver started at http://127.20.18.66:42573/ using document root <none> and password file <none>
I20250113 03:08:55.740643 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:55.740687 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:55.740731 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:55.740957 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.741740 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:55.742106 27157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.742203 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.742242 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.742297 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:55.768158 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:55.768332 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:55.768476 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:55.768674 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:55.768702 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.768731 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:55.768759 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.771312 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:37453
I20250113 03:08:55.771366 27220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:37453 every 8 connection(s)
I20250113 03:08:55.771627 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:08:55.772742 27224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:08:55.773512 27227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.774111 20553 server_base.cc:1034] running on GCE node
I20250113 03:08:55.774361 27221 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35427
I20250113 03:08:55.774415 27221 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:55.774508 27221 heartbeater.cc:510] Master 127.20.18.126:35427 requested a full tablet report, sending...
W20250113 03:08:55.774580 27225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:08:55.774755 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:08:55.774761 27016 ts_manager.cc:194] Registered new tserver with Master: 2aecc942a78d4bcfae37ff0aeaa27bb7 (127.20.18.66:37453)
W20250113 03:08:55.774794 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:08:55.774859 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737735774859 us; error 0 us; skew 500 ppm
I20250113 03:08:55.775240 27016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40802
I20250113 03:08:55.775548 20553 webserver.cc:458] Webserver started at http://127.20.18.67:35693/ using document root <none> and password file <none>
I20250113 03:08:55.775631 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:08:55.775674 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:08:55.775717 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:08:55.775985 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "932c3b2898fc416bb6f085f489e9a254"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.776906 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:08:55.777271 27232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.777371 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.777407 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "932c3b2898fc416bb6f085f489e9a254"
format_stamp: "Formatted at 2025-01-13 03:08:55 on dist-test-slave-d4dl"
I20250113 03:08:55.777448 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:08:55.787634 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:08:55.787778 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:08:55.787890 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:08:55.788053 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:08:55.788079 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.788103 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:08:55.788128 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.790930 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34117
I20250113 03:08:55.790973 27295 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34117 every 8 connection(s)
I20250113 03:08:55.793336 27296 heartbeater.cc:346] Connected to a master server at 127.20.18.126:35427
I20250113 03:08:55.793385 27296 heartbeater.cc:463] Registering TS with master...
I20250113 03:08:55.793465 27296 heartbeater.cc:510] Master 127.20.18.126:35427 requested a full tablet report, sending...
I20250113 03:08:55.793666 27016 ts_manager.cc:194] Registered new tserver with Master: 932c3b2898fc416bb6f085f489e9a254 (127.20.18.67:34117)
I20250113 03:08:55.793963 27016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40804
I20250113 03:08:55.794354 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003284269s
I20250113 03:08:55.794665 20553 test_util.cc:274] Using random seed: -1009819671
I20250113 03:08:55.798554 27016 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40818:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:08:55.798688 27016 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:08:55.802769 27185 tablet_service.cc:1467] Processing CreateTablet for tablet 25457b6caa73449a9f13b1ae174901d2 (DEFAULT_TABLE table=test-workload [id=b929350e60cb484883aa687be3d1022e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:55.802878 27185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25457b6caa73449a9f13b1ae174901d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:55.802985 27109 tablet_service.cc:1467] Processing CreateTablet for tablet 25457b6caa73449a9f13b1ae174901d2 (DEFAULT_TABLE table=test-workload [id=b929350e60cb484883aa687be3d1022e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:55.803020 27260 tablet_service.cc:1467] Processing CreateTablet for tablet 25457b6caa73449a9f13b1ae174901d2 (DEFAULT_TABLE table=test-workload [id=b929350e60cb484883aa687be3d1022e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:08:55.803081 27109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25457b6caa73449a9f13b1ae174901d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:55.803082 27260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25457b6caa73449a9f13b1ae174901d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:08:55.803927 27312 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: Bootstrap starting.
I20250113 03:08:55.804302 27314 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: Bootstrap starting.
I20250113 03:08:55.804508 27312 tablet_bootstrap.cc:654] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:55.804669 27313 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: Bootstrap starting.
I20250113 03:08:55.804714 27314 tablet_bootstrap.cc:654] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:55.805091 27313 tablet_bootstrap.cc:654] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: Neither blocks nor log segments found. Creating new log.
I20250113 03:08:55.805112 27312 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: No bootstrap required, opened a new log
I20250113 03:08:55.805152 27312 ts_tablet_manager.cc:1397] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:55.805284 27312 raft_consensus.cc:357] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.805327 27312 raft_consensus.cc:383] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:55.805351 27312 raft_consensus.cc:738] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aecc942a78d4bcfae37ff0aeaa27bb7, State: Initialized, Role: FOLLOWER
I20250113 03:08:55.805418 27312 consensus_queue.cc:260] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.805569 27312 ts_tablet_manager.cc:1428] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:08:55.805586 27221 heartbeater.cc:502] Master 127.20.18.126:35427 was elected leader, sending a full tablet report...
I20250113 03:08:55.805569 27314 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: No bootstrap required, opened a new log
I20250113 03:08:55.805661 27313 tablet_bootstrap.cc:492] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: No bootstrap required, opened a new log
I20250113 03:08:55.805703 27314 ts_tablet_manager.cc:1397] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:55.805714 27313 ts_tablet_manager.cc:1397] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:08:55.805827 27314 raft_consensus.cc:357] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.805845 27313 raft_consensus.cc:357] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.805903 27314 raft_consensus.cc:383] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:55.805970 27314 raft_consensus.cc:738] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85866f65084942ffb481710be83cd896, State: Initialized, Role: FOLLOWER
I20250113 03:08:55.805912 27313 raft_consensus.cc:383] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:08:55.806036 27313 raft_consensus.cc:738] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932c3b2898fc416bb6f085f489e9a254, State: Initialized, Role: FOLLOWER
I20250113 03:08:55.806080 27313 consensus_queue.cc:260] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.806094 27314 consensus_queue.cc:260] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.806245 27313 ts_tablet_manager.cc:1428] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:08:55.806278 27296 heartbeater.cc:502] Master 127.20.18.126:35427 was elected leader, sending a full tablet report...
I20250113 03:08:55.806279 27314 ts_tablet_manager.cc:1428] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:08:55.806380 27145 heartbeater.cc:502] Master 127.20.18.126:35427 was elected leader, sending a full tablet report...
I20250113 03:08:55.876477 27319 raft_consensus.cc:491] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:08:55.876603 27319 raft_consensus.cc:513] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.876771 27319 leader_election.cc:290] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aecc942a78d4bcfae37ff0aeaa27bb7 (127.20.18.66:37453), 85866f65084942ffb481710be83cd896 (127.20.18.65:35135)
I20250113 03:08:55.878947 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "932c3b2898fc416bb6f085f489e9a254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" is_pre_election: true
I20250113 03:08:55.879010 27195 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 932c3b2898fc416bb6f085f489e9a254 in term 0.
I20250113 03:08:55.879027 27119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "932c3b2898fc416bb6f085f489e9a254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85866f65084942ffb481710be83cd896" is_pre_election: true
I20250113 03:08:55.879118 27119 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 932c3b2898fc416bb6f085f489e9a254 in term 0.
I20250113 03:08:55.879158 27235 leader_election.cc:304] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aecc942a78d4bcfae37ff0aeaa27bb7, 932c3b2898fc416bb6f085f489e9a254; no voters:
I20250113 03:08:55.879230 27319 raft_consensus.cc:2798] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:08:55.879264 27319 raft_consensus.cc:491] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:08:55.879288 27319 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:55.879861 27319 raft_consensus.cc:513] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.879993 27319 leader_election.cc:290] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [CANDIDATE]: Term 1 election: Requested vote from peers 2aecc942a78d4bcfae37ff0aeaa27bb7 (127.20.18.66:37453), 85866f65084942ffb481710be83cd896 (127.20.18.65:35135)
I20250113 03:08:55.880125 27119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "932c3b2898fc416bb6f085f489e9a254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85866f65084942ffb481710be83cd896"
I20250113 03:08:55.880123 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "932c3b2898fc416bb6f085f489e9a254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7"
I20250113 03:08:55.880208 27119 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:55.880214 27195 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:08:55.880875 27119 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 932c3b2898fc416bb6f085f489e9a254 in term 1.
I20250113 03:08:55.880875 27195 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 932c3b2898fc416bb6f085f489e9a254 in term 1.
I20250113 03:08:55.881067 27235 leader_election.cc:304] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aecc942a78d4bcfae37ff0aeaa27bb7, 932c3b2898fc416bb6f085f489e9a254; no voters:
I20250113 03:08:55.881168 27319 raft_consensus.cc:2798] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:08:55.881302 27319 raft_consensus.cc:695] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 LEADER]: Becoming Leader. State: Replica: 932c3b2898fc416bb6f085f489e9a254, State: Running, Role: LEADER
I20250113 03:08:55.881367 27319 consensus_queue.cc:237] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:08:55.881809 27015 catalog_manager.cc:5526] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 932c3b2898fc416bb6f085f489e9a254 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "932c3b2898fc416bb6f085f489e9a254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } health_report { overall_health: HEALTHY } } }
I20250113 03:08:55.896512 27119 raft_consensus.cc:1270] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 1 FOLLOWER]: Refusing update from remote peer 932c3b2898fc416bb6f085f489e9a254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:55.896546 27195 raft_consensus.cc:1270] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 1 FOLLOWER]: Refusing update from remote peer 932c3b2898fc416bb6f085f489e9a254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:08:55.896749 27322 consensus_queue.cc:1035] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:08:55.896833 27319 consensus_queue.cc:1035] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:06.462265 27363 raft_consensus.cc:988] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: : Instructing follower 85866f65084942ffb481710be83cd896 to start an election
I20250113 03:09:06.462340 27363 raft_consensus.cc:1076] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 LEADER]: Signalling peer 85866f65084942ffb481710be83cd896 to start an election
I20250113 03:09:06.462491 27119 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2"
dest_uuid: "85866f65084942ffb481710be83cd896"
from {username='slave'} at 127.0.0.1:56220
I20250113 03:09:06.462569 27119 raft_consensus.cc:491] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:09:06.462600 27119 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:06.463186 27119 raft_consensus.cc:513] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:09:06.463325 27119 leader_election.cc:290] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [CANDIDATE]: Term 2 election: Requested vote from peers 2aecc942a78d4bcfae37ff0aeaa27bb7 (127.20.18.66:37453), 932c3b2898fc416bb6f085f489e9a254 (127.20.18.67:34117)
I20250113 03:09:06.465770 27270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "85866f65084942ffb481710be83cd896" candidate_term: 2 candidate_status { last_received { term: 1 index: 15575 } } ignore_live_leader: true dest_uuid: "932c3b2898fc416bb6f085f489e9a254"
I20250113 03:09:06.465834 27270 raft_consensus.cc:3049] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:09:06.465829 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25457b6caa73449a9f13b1ae174901d2" candidate_uuid: "85866f65084942ffb481710be83cd896" candidate_term: 2 candidate_status { last_received { term: 1 index: 15575 } } ignore_live_leader: true dest_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7"
I20250113 03:09:06.465914 27270 raft_consensus.cc:738] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 932c3b2898fc416bb6f085f489e9a254, State: Running, Role: LEADER
I20250113 03:09:06.465934 27195 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:06.465963 27270 consensus_queue.cc:260] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15575, Committed index: 15575, Last appended: 1.15575, Last appended by leader: 15575, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:09:06.466053 27270 raft_consensus.cc:3054] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:06.466508 27195 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85866f65084942ffb481710be83cd896 in term 2.
I20250113 03:09:06.466593 27270 raft_consensus.cc:2463] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85866f65084942ffb481710be83cd896 in term 2.
I20250113 03:09:06.466683 27084 leader_election.cc:304] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aecc942a78d4bcfae37ff0aeaa27bb7, 85866f65084942ffb481710be83cd896; no voters:
I20250113 03:09:06.466823 27375 raft_consensus.cc:2798] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:09:06.466928 27375 raft_consensus.cc:695] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 2 LEADER]: Becoming Leader. State: Replica: 85866f65084942ffb481710be83cd896, State: Running, Role: LEADER
I20250113 03:09:06.466979 27375 consensus_queue.cc:237] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15575, Committed index: 15575, Last appended: 1.15575, Last appended by leader: 15575, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } }
I20250113 03:09:06.467372 27009 catalog_manager.cc:5526] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 reported cstate change: term changed from 1 to 2, leader changed from 932c3b2898fc416bb6f085f489e9a254 (127.20.18.67) to 85866f65084942ffb481710be83cd896 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "85866f65084942ffb481710be83cd896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85866f65084942ffb481710be83cd896" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 35135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 } health_report { overall_health: UNKNOWN } } }
W20250113 03:09:06.480198 27390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38462: Illegal state: Transaction hasn't been successfully started
W20250113 03:09:06.480604 27390 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38462: Illegal state: Transaction metadata for transaction 1 does not exist
I20250113 03:09:06.481119 27270 raft_consensus.cc:1270] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 2 FOLLOWER]: Refusing update from remote peer 85866f65084942ffb481710be83cd896: Log matching property violated. Preceding OpId in replica: term: 1 index: 15575. Preceding OpId from leader: term: 2 index: 15577. (index mismatch)
I20250113 03:09:06.481129 27195 raft_consensus.cc:1270] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 2 FOLLOWER]: Refusing update from remote peer 85866f65084942ffb481710be83cd896: Log matching property violated. Preceding OpId in replica: term: 1 index: 15575. Preceding OpId from leader: term: 2 index: 15577. (index mismatch)
I20250113 03:09:06.481300 27376 consensus_queue.cc:1035] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "932c3b2898fc416bb6f085f489e9a254" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15576, Last known committed idx: 15575, Time since last communication: 0.000s
I20250113 03:09:06.481379 27375 consensus_queue.cc:1035] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aecc942a78d4bcfae37ff0aeaa27bb7" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15576, Last known committed idx: 15575, Time since last communication: 0.000s
I20250113 03:09:06.483860 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:09:06.486130 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:06.486214 20553 tablet_replica.cc:331] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896: stopping tablet replica
I20250113 03:09:06.486268 20553 raft_consensus.cc:2238] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:09:06.486310 20553 raft_consensus.cc:2267] T 25457b6caa73449a9f13b1ae174901d2 P 85866f65084942ffb481710be83cd896 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:06.499934 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:09:06.500687 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:09:06.503676 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:06.503795 20553 tablet_replica.cc:331] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7: stopping tablet replica
I20250113 03:09:06.503852 20553 raft_consensus.cc:2238] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:06.503896 20553 raft_consensus.cc:2267] T 25457b6caa73449a9f13b1ae174901d2 P 2aecc942a78d4bcfae37ff0aeaa27bb7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:06.525441 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:09:06.526176 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:09:06.528581 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:06.528683 20553 tablet_replica.cc:331] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254: stopping tablet replica
I20250113 03:09:06.528748 20553 raft_consensus.cc:2238] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:06.528797 20553 raft_consensus.cc:2267] T 25457b6caa73449a9f13b1ae174901d2 P 932c3b2898fc416bb6f085f489e9a254 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:06.542062 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:09:06.542853 20553 master.cc:537] Master@127.20.18.126:35427 shutting down...
I20250113 03:09:06.545097 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:09:06.545162 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:06.545183 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 90bb1fc13245418d84477704f1c9db51: stopping tablet replica
I20250113 03:09:06.556785 20553 master.cc:559] Master@127.20.18.126:35427 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun (10928 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRepeatCalls
I20250113 03:09:06.575656 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:33063
I20250113 03:09:06.575809 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:06.577469 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.577644 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.578289 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:06.579066 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:06.579169 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:06.579195 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:06.579214 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737746579214 us; error 0 us; skew 500 ppm
I20250113 03:09:06.579758 20553 webserver.cc:458] Webserver started at http://127.20.18.126:33057/ using document root <none> and password file <none>
I20250113 03:09:06.579833 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:06.579869 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:06.579911 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:06.580157 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "06ef25b742cf4f799c9256758db1d14e"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.580991 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:06.581384 27406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.581491 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.581527 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "06ef25b742cf4f799c9256758db1d14e"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.581570 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:06.594527 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:06.594664 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:06.596921 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:33063
I20250113 03:09:06.607973 27458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:33063 every 8 connection(s)
I20250113 03:09:06.608108 27459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:06.609128 27459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e: Bootstrap starting.
I20250113 03:09:06.609449 27459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:06.609922 27459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e: No bootstrap required, opened a new log
I20250113 03:09:06.610035 27459 raft_consensus.cc:357] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER }
I20250113 03:09:06.610087 27459 raft_consensus.cc:383] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:06.610101 27459 raft_consensus.cc:738] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ef25b742cf4f799c9256758db1d14e, State: Initialized, Role: FOLLOWER
I20250113 03:09:06.610154 27459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER }
I20250113 03:09:06.610193 27459 raft_consensus.cc:397] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:09:06.610211 27459 raft_consensus.cc:491] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:09:06.610239 27459 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:06.610702 27459 raft_consensus.cc:513] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER }
I20250113 03:09:06.610752 27459 leader_election.cc:304] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ef25b742cf4f799c9256758db1d14e; no voters:
I20250113 03:09:06.610857 27459 leader_election.cc:290] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:09:06.610896 27462 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:06.611002 27462 raft_consensus.cc:695] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 1 LEADER]: Becoming Leader. State: Replica: 06ef25b742cf4f799c9256758db1d14e, State: Running, Role: LEADER
I20250113 03:09:06.611001 27459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:09:06.611060 27462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER }
I20250113 03:09:06.611228 27462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ef25b742cf4f799c9256758db1d14e. Latest consensus state: current_term: 1 leader_uuid: "06ef25b742cf4f799c9256758db1d14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER } }
I20250113 03:09:06.611344 27462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:06.611268 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ef25b742cf4f799c9256758db1d14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef25b742cf4f799c9256758db1d14e" member_type: VOTER } }
I20250113 03:09:06.611630 27467 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:09:06.611649 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:06.611797 27467 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:09:06.612294 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:09:06.612355 27467 catalog_manager.cc:1348] Generated new cluster ID: fd0af70ecd1f4aee8bdca59358f293e8
I20250113 03:09:06.612385 27467 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:09:06.621372 27467 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:09:06.621728 27467 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:09:06.625592 27467 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e: Generated new TSK 0
I20250113 03:09:06.625654 27467 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:09:06.627921 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:06.629153 27480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.629894 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.630594 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:06.631206 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:06.631296 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:06.631330 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:06.631345 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737746631345 us; error 0 us; skew 500 ppm
I20250113 03:09:06.631955 20553 webserver.cc:458] Webserver started at http://127.20.18.65:33461/ using document root <none> and password file <none>
I20250113 03:09:06.632021 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:06.632050 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:06.632076 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:06.632278 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "d69734f12c3840b1bf598d6cade41bbf"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.633123 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:06.633513 27488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.633630 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.633674 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "d69734f12c3840b1bf598d6cade41bbf"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.633706 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:06.652806 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:06.652940 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:06.653044 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:06.653210 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:06.653234 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.653263 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:06.653285 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.655761 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:37487
I20250113 03:09:06.655812 27551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:37487 every 8 connection(s)
I20250113 03:09:06.656050 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:06.657145 27556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.657832 27557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.658565 27559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:06.659024 27552 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33063
I20250113 03:09:06.659082 27552 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:06.659183 27552 heartbeater.cc:510] Master 127.20.18.126:33063 requested a full tablet report, sending...
I20250113 03:09:06.659199 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:06.659300 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:06.659343 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:06.659358 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737746659357 us; error 0 us; skew 500 ppm
I20250113 03:09:06.659461 27423 ts_manager.cc:194] Registered new tserver with Master: d69734f12c3840b1bf598d6cade41bbf (127.20.18.65:37487)
I20250113 03:09:06.660037 20553 webserver.cc:458] Webserver started at http://127.20.18.66:42383/ using document root <none> and password file <none>
I20250113 03:09:06.660112 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:06.660147 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:06.660187 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:06.660204 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48556
I20250113 03:09:06.660476 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "45f610ca906d4240893124f996a075db"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.661312 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:06.661675 27564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.661775 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.661805 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "45f610ca906d4240893124f996a075db"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.661828 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:06.667160 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:06.667272 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:06.667358 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:06.667471 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:06.667492 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.667521 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:06.667538 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.669986 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:32783
I20250113 03:09:06.670011 27627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:32783 every 8 connection(s)
I20250113 03:09:06.670282 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:06.671922 27634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.672634 27632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:06.672648 27631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:06.673035 27628 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33063
I20250113 03:09:06.673081 27628 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:06.673161 27628 heartbeater.cc:510] Master 127.20.18.126:33063 requested a full tablet report, sending...
I20250113 03:09:06.673208 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:06.673336 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:09:06.673344 27423 ts_manager.cc:194] Registered new tserver with Master: 45f610ca906d4240893124f996a075db (127.20.18.66:32783)
W20250113 03:09:06.673378 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:06.673394 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737746673395 us; error 0 us; skew 500 ppm
I20250113 03:09:06.673722 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48566
I20250113 03:09:06.673964 20553 webserver.cc:458] Webserver started at http://127.20.18.67:35901/ using document root <none> and password file <none>
I20250113 03:09:06.674041 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:06.674093 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:06.674142 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:06.674358 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "93c503d208fd48d1ae66ad86ebc7f434"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.675069 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:06.675418 27639 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.675519 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.675565 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "93c503d208fd48d1ae66ad86ebc7f434"
format_stamp: "Formatted at 2025-01-13 03:09:06 on dist-test-slave-d4dl"
I20250113 03:09:06.675591 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:06.693420 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:06.693558 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:06.693667 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:06.693816 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:06.693840 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.693856 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:06.693879 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.696738 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:43575
I20250113 03:09:06.696794 27702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:43575 every 8 connection(s)
I20250113 03:09:06.699103 27703 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33063
I20250113 03:09:06.699152 27703 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:06.699249 27703 heartbeater.cc:510] Master 127.20.18.126:33063 requested a full tablet report, sending...
I20250113 03:09:06.699424 27423 ts_manager.cc:194] Registered new tserver with Master: 93c503d208fd48d1ae66ad86ebc7f434 (127.20.18.67:43575)
I20250113 03:09:06.699690 27423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48578
I20250113 03:09:06.700174 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003261103s
I20250113 03:09:06.700455 20553 test_util.cc:274] Using random seed: -998913881
I20250113 03:09:06.704417 27423 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48590:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:09:06.704571 27423 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:09:06.708508 27516 tablet_service.cc:1467] Processing CreateTablet for tablet ce70a2702cfe4c1cbdb16da4add1e976 (DEFAULT_TABLE table=test-workload [id=41cc625c1e224579a14487a8d33273d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:06.708647 27516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce70a2702cfe4c1cbdb16da4add1e976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:06.709023 27667 tablet_service.cc:1467] Processing CreateTablet for tablet ce70a2702cfe4c1cbdb16da4add1e976 (DEFAULT_TABLE table=test-workload [id=41cc625c1e224579a14487a8d33273d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:06.709134 27667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce70a2702cfe4c1cbdb16da4add1e976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:06.709883 27718 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: Bootstrap starting.
I20250113 03:09:06.709960 27592 tablet_service.cc:1467] Processing CreateTablet for tablet ce70a2702cfe4c1cbdb16da4add1e976 (DEFAULT_TABLE table=test-workload [id=41cc625c1e224579a14487a8d33273d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:06.710057 27592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce70a2702cfe4c1cbdb16da4add1e976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:06.710292 27718 tablet_bootstrap.cc:654] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:06.710824 27718 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: No bootstrap required, opened a new log
I20250113 03:09:06.710872 27718 ts_tablet_manager.cc:1397] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:06.711009 27718 raft_consensus.cc:357] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.711071 27718 raft_consensus.cc:383] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:06.711102 27718 raft_consensus.cc:738] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d69734f12c3840b1bf598d6cade41bbf, State: Initialized, Role: FOLLOWER
I20250113 03:09:06.711180 27718 consensus_queue.cc:260] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.711313 27721 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: Bootstrap starting.
I20250113 03:09:06.711337 27718 ts_tablet_manager.cc:1428] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:06.711403 27552 heartbeater.cc:502] Master 127.20.18.126:33063 was elected leader, sending a full tablet report...
I20250113 03:09:06.711505 27719 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: Bootstrap starting.
I20250113 03:09:06.711819 27721 tablet_bootstrap.cc:654] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:06.711936 27719 tablet_bootstrap.cc:654] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:06.712414 27721 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: No bootstrap required, opened a new log
I20250113 03:09:06.712457 27721 ts_tablet_manager.cc:1397] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:06.712567 27719 tablet_bootstrap.cc:492] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: No bootstrap required, opened a new log
I20250113 03:09:06.712615 27719 ts_tablet_manager.cc:1397] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:06.712626 27721 raft_consensus.cc:357] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.712693 27721 raft_consensus.cc:383] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:06.712718 27721 raft_consensus.cc:738] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f610ca906d4240893124f996a075db, State: Initialized, Role: FOLLOWER
I20250113 03:09:06.712743 27719 raft_consensus.cc:357] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.712803 27719 raft_consensus.cc:383] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:06.712783 27721 consensus_queue.cc:260] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.712836 27719 raft_consensus.cc:738] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c503d208fd48d1ae66ad86ebc7f434, State: Initialized, Role: FOLLOWER
I20250113 03:09:06.712872 27719 consensus_queue.cc:260] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.712950 27721 ts_tablet_manager.cc:1428] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 03:09:06.712980 27628 heartbeater.cc:502] Master 127.20.18.126:33063 was elected leader, sending a full tablet report...
I20250113 03:09:06.713018 27703 heartbeater.cc:502] Master 127.20.18.126:33063 was elected leader, sending a full tablet report...
I20250113 03:09:06.713018 27719 ts_tablet_manager.cc:1428] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:06.754142 27726 raft_consensus.cc:491] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:09:06.754230 27726 raft_consensus.cc:513] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.754432 27726 leader_election.cc:290] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d69734f12c3840b1bf598d6cade41bbf (127.20.18.65:37487), 45f610ca906d4240893124f996a075db (127.20.18.66:32783)
I20250113 03:09:06.757607 27602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "93c503d208fd48d1ae66ad86ebc7f434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f610ca906d4240893124f996a075db" is_pre_election: true
I20250113 03:09:06.757607 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "93c503d208fd48d1ae66ad86ebc7f434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69734f12c3840b1bf598d6cade41bbf" is_pre_election: true
I20250113 03:09:06.757730 27602 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c503d208fd48d1ae66ad86ebc7f434 in term 0.
I20250113 03:09:06.757735 27526 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c503d208fd48d1ae66ad86ebc7f434 in term 0.
I20250113 03:09:06.757889 27640 leader_election.cc:304] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f610ca906d4240893124f996a075db, 93c503d208fd48d1ae66ad86ebc7f434; no voters:
I20250113 03:09:06.757995 27726 raft_consensus.cc:2798] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:09:06.758052 27726 raft_consensus.cc:491] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:09:06.758074 27726 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:06.758679 27726 raft_consensus.cc:513] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.758795 27726 leader_election.cc:290] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [CANDIDATE]: Term 1 election: Requested vote from peers d69734f12c3840b1bf598d6cade41bbf (127.20.18.65:37487), 45f610ca906d4240893124f996a075db (127.20.18.66:32783)
I20250113 03:09:06.758972 27526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "93c503d208fd48d1ae66ad86ebc7f434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69734f12c3840b1bf598d6cade41bbf"
I20250113 03:09:06.759038 27526 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:06.759043 27602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "93c503d208fd48d1ae66ad86ebc7f434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f610ca906d4240893124f996a075db"
I20250113 03:09:06.759101 27602 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:06.759733 27602 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c503d208fd48d1ae66ad86ebc7f434 in term 1.
I20250113 03:09:06.759733 27526 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c503d208fd48d1ae66ad86ebc7f434 in term 1.
I20250113 03:09:06.759909 27640 leader_election.cc:304] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f610ca906d4240893124f996a075db, 93c503d208fd48d1ae66ad86ebc7f434; no voters:
I20250113 03:09:06.759994 27726 raft_consensus.cc:2798] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:06.760133 27726 raft_consensus.cc:695] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 LEADER]: Becoming Leader. State: Replica: 93c503d208fd48d1ae66ad86ebc7f434, State: Running, Role: LEADER
I20250113 03:09:06.760185 27726 consensus_queue.cc:237] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:06.760676 27422 catalog_manager.cc:5526] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93c503d208fd48d1ae66ad86ebc7f434 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "93c503d208fd48d1ae66ad86ebc7f434" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:06.772677 27602 raft_consensus.cc:1270] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 1 FOLLOWER]: Refusing update from remote peer 93c503d208fd48d1ae66ad86ebc7f434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:06.772701 27526 raft_consensus.cc:1270] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 1 FOLLOWER]: Refusing update from remote peer 93c503d208fd48d1ae66ad86ebc7f434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:06.772904 27728 consensus_queue.cc:1035] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:06.773000 27726 consensus_queue.cc:1035] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:06.774417 27736 mvcc.cc:204] Tried to move back new op lower bound from 7113677810781106176 to 7113677810730151936. Current Snapshot: MvccSnapshot[applied={T|T < 7113677810779766784}]
I20250113 03:09:17.281577 27778 raft_consensus.cc:988] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: : Instructing follower d69734f12c3840b1bf598d6cade41bbf to start an election
I20250113 03:09:17.281654 27778 raft_consensus.cc:1076] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 LEADER]: Signalling peer d69734f12c3840b1bf598d6cade41bbf to start an election
I20250113 03:09:17.281800 27526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976"
dest_uuid: "d69734f12c3840b1bf598d6cade41bbf"
from {username='slave'} at 127.0.0.1:37926
I20250113 03:09:17.281857 27526 raft_consensus.cc:491] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:09:17.281876 27526 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:17.282451 27526 raft_consensus.cc:513] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:17.282572 27526 leader_election.cc:290] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [CANDIDATE]: Term 2 election: Requested vote from peers 93c503d208fd48d1ae66ad86ebc7f434 (127.20.18.67:43575), 45f610ca906d4240893124f996a075db (127.20.18.66:32783)
I20250113 03:09:17.285699 27677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "d69734f12c3840b1bf598d6cade41bbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 15718 } } ignore_live_leader: true dest_uuid: "93c503d208fd48d1ae66ad86ebc7f434"
I20250113 03:09:17.285703 27602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce70a2702cfe4c1cbdb16da4add1e976" candidate_uuid: "d69734f12c3840b1bf598d6cade41bbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 15718 } } ignore_live_leader: true dest_uuid: "45f610ca906d4240893124f996a075db"
I20250113 03:09:17.285776 27677 raft_consensus.cc:3049] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:09:17.285786 27602 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:17.285812 27677 raft_consensus.cc:738] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93c503d208fd48d1ae66ad86ebc7f434, State: Running, Role: LEADER
I20250113 03:09:17.285873 27677 consensus_queue.cc:260] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15718, Committed index: 15718, Last appended: 1.15718, Last appended by leader: 15718, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:17.285960 27677 raft_consensus.cc:3054] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:17.286346 27602 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d69734f12c3840b1bf598d6cade41bbf in term 2.
I20250113 03:09:17.286546 27489 leader_election.cc:304] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f610ca906d4240893124f996a075db, d69734f12c3840b1bf598d6cade41bbf; no voters:
I20250113 03:09:17.286551 27677 raft_consensus.cc:2463] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d69734f12c3840b1bf598d6cade41bbf in term 2.
I20250113 03:09:17.286736 27784 raft_consensus.cc:2798] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:09:17.286844 27784 raft_consensus.cc:695] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 2 LEADER]: Becoming Leader. State: Replica: d69734f12c3840b1bf598d6cade41bbf, State: Running, Role: LEADER
I20250113 03:09:17.286899 27784 consensus_queue.cc:237] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15718, Committed index: 15718, Last appended: 1.15718, Last appended by leader: 15718, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } }
I20250113 03:09:17.287282 27422 catalog_manager.cc:5526] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf reported cstate change: term changed from 1 to 2, leader changed from 93c503d208fd48d1ae66ad86ebc7f434 (127.20.18.67) to d69734f12c3840b1bf598d6cade41bbf (127.20.18.65). New cstate: current_term: 2 leader_uuid: "d69734f12c3840b1bf598d6cade41bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d69734f12c3840b1bf598d6cade41bbf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:17.301285 27677 raft_consensus.cc:1270] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 2 FOLLOWER]: Refusing update from remote peer d69734f12c3840b1bf598d6cade41bbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 15718. Preceding OpId from leader: term: 2 index: 15720. (index mismatch)
I20250113 03:09:17.301285 27602 raft_consensus.cc:1270] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 2 FOLLOWER]: Refusing update from remote peer d69734f12c3840b1bf598d6cade41bbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 15718. Preceding OpId from leader: term: 2 index: 15720. (index mismatch)
I20250113 03:09:17.301499 27785 consensus_queue.cc:1035] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c503d208fd48d1ae66ad86ebc7f434" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15719, Last known committed idx: 15718, Time since last communication: 0.000s
I20250113 03:09:17.301595 27784 consensus_queue.cc:1035] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f610ca906d4240893124f996a075db" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15719, Last known committed idx: 15718, Time since last communication: 0.000s
W20250113 03:09:17.302878 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction 1 already open
W20250113 03:09:17.304037 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction 1 commit already in progress
W20250113 03:09:17.305142 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction metadata for transaction 1 already set
W20250113 03:09:17.306327 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction 2 already open
W20250113 03:09:17.307476 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction 2 commit already in progress
W20250113 03:09:17.308669 27798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46458: Illegal state: Transaction metadata for transaction 2 already set
I20250113 03:09:17.308974 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:09:17.310940 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:17.311026 20553 tablet_replica.cc:331] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf: stopping tablet replica
I20250113 03:09:17.311071 20553 raft_consensus.cc:2238] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:09:17.311108 20553 raft_consensus.cc:2267] T ce70a2702cfe4c1cbdb16da4add1e976 P d69734f12c3840b1bf598d6cade41bbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:17.324646 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:09:17.325237 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:09:17.328383 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:17.328514 20553 tablet_replica.cc:331] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db: stopping tablet replica
I20250113 03:09:17.328588 20553 raft_consensus.cc:2238] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:17.328625 20553 raft_consensus.cc:2267] T ce70a2702cfe4c1cbdb16da4add1e976 P 45f610ca906d4240893124f996a075db [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:17.347229 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:09:17.347874 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:09:17.350652 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:17.350781 20553 tablet_replica.cc:331] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434: stopping tablet replica
I20250113 03:09:17.350824 20553 raft_consensus.cc:2238] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:17.350860 20553 raft_consensus.cc:2267] T ce70a2702cfe4c1cbdb16da4add1e976 P 93c503d208fd48d1ae66ad86ebc7f434 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:17.372306 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:09:17.373063 20553 master.cc:537] Master@127.20.18.126:33063 shutting down...
I20250113 03:09:17.375105 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:09:17.375159 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:17.375183 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 06ef25b742cf4f799c9256758db1d14e: stopping tablet replica
I20250113 03:09:17.386633 20553 master.cc:559] Master@127.20.18.126:33063 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRepeatCalls (10828 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority
I20250113 03:09:17.404199 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:40849
I20250113 03:09:17.404327 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:17.405854 27806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.406563 27809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.407073 27807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:17.407958 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:17.408053 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:17.408079 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:17.408090 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737757408091 us; error 0 us; skew 500 ppm
I20250113 03:09:17.408672 20553 webserver.cc:458] Webserver started at http://127.20.18.126:45307/ using document root <none> and password file <none>
I20250113 03:09:17.408730 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:17.408767 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:17.408810 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:17.409065 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "2dea1f47ce904d03bbdbe73b653e5b56"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.409785 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:17.410128 27814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.410235 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.410274 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "2dea1f47ce904d03bbdbe73b653e5b56"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.410311 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:17.421058 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:17.421182 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:17.423151 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:40849
I20250113 03:09:17.424599 27866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:40849 every 8 connection(s)
I20250113 03:09:17.424691 27867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:17.426244 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56: Bootstrap starting.
I20250113 03:09:17.426510 27867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:17.426888 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56: No bootstrap required, opened a new log
I20250113 03:09:17.426975 27867 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER }
I20250113 03:09:17.427031 27867 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:17.427043 27867 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dea1f47ce904d03bbdbe73b653e5b56, State: Initialized, Role: FOLLOWER
I20250113 03:09:17.427075 27867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER }
I20250113 03:09:17.427117 27867 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:09:17.427137 27867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:09:17.427156 27867 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:17.427568 27867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER }
I20250113 03:09:17.427613 27867 leader_election.cc:304] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dea1f47ce904d03bbdbe73b653e5b56; no voters:
I20250113 03:09:17.427685 27867 leader_election.cc:290] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:09:17.427747 27870 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:17.427819 27867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:09:17.427870 27870 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 1 LEADER]: Becoming Leader. State: Replica: 2dea1f47ce904d03bbdbe73b653e5b56, State: Running, Role: LEADER
I20250113 03:09:17.427913 27870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER }
I20250113 03:09:17.428059 27870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dea1f47ce904d03bbdbe73b653e5b56. Latest consensus state: current_term: 1 leader_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER } }
I20250113 03:09:17.428092 27871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea1f47ce904d03bbdbe73b653e5b56" member_type: VOTER } }
I20250113 03:09:17.428174 27870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:17.428217 27871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:17.428459 27877 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:09:17.428638 27877 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:09:17.429044 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:09:17.429177 27877 catalog_manager.cc:1348] Generated new cluster ID: ae5cad8f08814c28b48e3e419917c620
I20250113 03:09:17.429220 27877 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:09:17.443663 27877 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:09:17.443970 27877 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:09:17.446870 27877 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56: Generated new TSK 0
I20250113 03:09:17.446931 27877 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:09:17.460780 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:17.462138 27888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.462826 27891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:17.463411 20553 server_base.cc:1034] running on GCE node
W20250113 03:09:17.464105 27889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:17.464212 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:17.464241 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:17.464258 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737757464258 us; error 0 us; skew 500 ppm
I20250113 03:09:17.464815 20553 webserver.cc:458] Webserver started at http://127.20.18.65:33945/ using document root <none> and password file <none>
I20250113 03:09:17.464887 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:17.464924 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:17.464964 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:17.465169 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "b0d53b1c670a4095a7c570f6585f7c50"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.465919 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:17.466291 27896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.466415 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.466452 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "b0d53b1c670a4095a7c570f6585f7c50"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.466493 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:17.481983 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:17.482115 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:17.482223 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:17.482374 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:17.482398 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.482431 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:17.482455 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.484824 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:39997
I20250113 03:09:17.485312 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:09:17.485407 27959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:39997 every 8 connection(s)
W20250113 03:09:17.486312 27962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.487104 27967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.487843 27963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:17.488358 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:17.488448 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:17.488485 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:17.488504 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737757488504 us; error 0 us; skew 500 ppm
I20250113 03:09:17.488570 27960 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40849
I20250113 03:09:17.488623 27960 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:17.488730 27960 heartbeater.cc:510] Master 127.20.18.126:40849 requested a full tablet report, sending...
I20250113 03:09:17.488937 27831 ts_manager.cc:194] Registered new tserver with Master: b0d53b1c670a4095a7c570f6585f7c50 (127.20.18.65:39997)
I20250113 03:09:17.489091 20553 webserver.cc:458] Webserver started at http://127.20.18.66:39235/ using document root <none> and password file <none>
I20250113 03:09:17.489156 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:17.489194 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:17.489230 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:17.489467 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "48403f235703436da76148a97acd1868"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.489542 27831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49890
I20250113 03:09:17.490203 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:17.490551 27972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.490629 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.490671 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "48403f235703436da76148a97acd1868"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.490707 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:17.499769 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:17.499891 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:17.499997 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:17.500137 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:17.500159 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.500185 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:17.500209 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.502507 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:43507
I20250113 03:09:17.502557 28035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:43507 every 8 connection(s)
I20250113 03:09:17.502792 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:17.503747 28039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.504362 28040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:17.504989 28042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:17.505188 28036 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40849
I20250113 03:09:17.505230 28036 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:17.505291 28036 heartbeater.cc:510] Master 127.20.18.126:40849 requested a full tablet report, sending...
I20250113 03:09:17.505460 27831 ts_manager.cc:194] Registered new tserver with Master: 48403f235703436da76148a97acd1868 (127.20.18.66:43507)
I20250113 03:09:17.505498 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:17.505645 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:17.505676 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:17.505687 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737757505688 us; error 0 us; skew 500 ppm
I20250113 03:09:17.505779 27831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49892
I20250113 03:09:17.506232 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43223/ using document root <none> and password file <none>
I20250113 03:09:17.506284 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:17.506323 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:17.506405 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:17.506601 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "8015254d551147de8b1379c32c88ac8f"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.507313 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:17.507694 28047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.507803 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.507839 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "8015254d551147de8b1379c32c88ac8f"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:17.507875 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:17.528220 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:17.528424 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:17.528584 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:17.528744 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:17.528766 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.528793 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:17.528818 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:17.531478 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:37193
I20250113 03:09:17.531530 28110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:37193 every 8 connection(s)
I20250113 03:09:17.533676 28111 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40849
I20250113 03:09:17.533717 28111 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:17.533794 28111 heartbeater.cc:510] Master 127.20.18.126:40849 requested a full tablet report, sending...
I20250113 03:09:17.533994 27831 ts_manager.cc:194] Registered new tserver with Master: 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193)
I20250113 03:09:17.534286 27831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49898
I20250113 03:09:17.534881 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003247259s
I20250113 03:09:17.535172 20553 test_util.cc:274] Using random seed: -988079164
I20250113 03:09:17.538988 27831 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49910:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:09:17.539134 27831 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:09:17.542888 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 92f5be83c9ee4c8eba79ac1b661edaa3 (DEFAULT_TABLE table=test-workload [id=fc54f4908ec344e999084c3eb720bcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:17.542986 28000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f5be83c9ee4c8eba79ac1b661edaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:17.543056 28075 tablet_service.cc:1467] Processing CreateTablet for tablet 92f5be83c9ee4c8eba79ac1b661edaa3 (DEFAULT_TABLE table=test-workload [id=fc54f4908ec344e999084c3eb720bcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:17.543155 28075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f5be83c9ee4c8eba79ac1b661edaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:17.544198 27924 tablet_service.cc:1467] Processing CreateTablet for tablet 92f5be83c9ee4c8eba79ac1b661edaa3 (DEFAULT_TABLE table=test-workload [id=fc54f4908ec344e999084c3eb720bcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:17.544322 28127 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap starting.
I20250113 03:09:17.544194 28126 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap starting.
I20250113 03:09:17.544399 27924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f5be83c9ee4c8eba79ac1b661edaa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:17.544739 28127 tablet_bootstrap.cc:654] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:17.544790 28126 tablet_bootstrap.cc:654] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:17.545248 28127 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: No bootstrap required, opened a new log
I20250113 03:09:17.545302 28127 ts_tablet_manager.cc:1397] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:17.545337 28126 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: No bootstrap required, opened a new log
I20250113 03:09:17.545393 28126 ts_tablet_manager.cc:1397] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:17.545446 28127 raft_consensus.cc:357] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.545498 28127 raft_consensus.cc:383] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:17.545518 28127 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8015254d551147de8b1379c32c88ac8f, State: Initialized, Role: FOLLOWER
I20250113 03:09:17.545531 28126 raft_consensus.cc:357] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.545565 28127 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.545626 28126 raft_consensus.cc:383] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:17.545650 28126 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48403f235703436da76148a97acd1868, State: Initialized, Role: FOLLOWER
I20250113 03:09:17.545712 28126 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.545732 28127 ts_tablet_manager.cc:1428] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 03:09:17.545807 28111 heartbeater.cc:502] Master 127.20.18.126:40849 was elected leader, sending a full tablet report...
I20250113 03:09:17.545835 28126 ts_tablet_manager.cc:1428] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250113 03:09:17.545877 28036 heartbeater.cc:502] Master 127.20.18.126:40849 was elected leader, sending a full tablet report...
I20250113 03:09:17.545732 28130 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: Bootstrap starting.
I20250113 03:09:17.546424 28130 tablet_bootstrap.cc:654] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:17.546943 28130 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: No bootstrap required, opened a new log
I20250113 03:09:17.546993 28130 ts_tablet_manager.cc:1397] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:17.547117 28130 raft_consensus.cc:357] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.547158 28130 raft_consensus.cc:383] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:17.547170 28130 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d53b1c670a4095a7c570f6585f7c50, State: Initialized, Role: FOLLOWER
I20250113 03:09:17.547216 28130 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.547351 28130 ts_tablet_manager.cc:1428] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:17.547379 27960 heartbeater.cc:502] Master 127.20.18.126:40849 was elected leader, sending a full tablet report...
I20250113 03:09:17.586309 28132 raft_consensus.cc:491] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:09:17.586376 28132 raft_consensus.cc:513] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.586522 28132 leader_election.cc:290] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d53b1c670a4095a7c570f6585f7c50 (127.20.18.65:39997), 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193)
I20250113 03:09:17.589387 28085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "48403f235703436da76148a97acd1868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8015254d551147de8b1379c32c88ac8f" is_pre_election: true
I20250113 03:09:17.589386 27934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "48403f235703436da76148a97acd1868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d53b1c670a4095a7c570f6585f7c50" is_pre_election: true
I20250113 03:09:17.589504 28085 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48403f235703436da76148a97acd1868 in term 0.
I20250113 03:09:17.589540 27934 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48403f235703436da76148a97acd1868 in term 0.
I20250113 03:09:17.589671 27976 leader_election.cc:304] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48403f235703436da76148a97acd1868, 8015254d551147de8b1379c32c88ac8f; no voters:
I20250113 03:09:17.589754 28132 raft_consensus.cc:2798] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:09:17.589792 28132 raft_consensus.cc:491] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:09:17.589820 28132 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:17.590315 28132 raft_consensus.cc:513] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.590420 28132 leader_election.cc:290] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [CANDIDATE]: Term 1 election: Requested vote from peers b0d53b1c670a4095a7c570f6585f7c50 (127.20.18.65:39997), 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193)
I20250113 03:09:17.590531 28085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "48403f235703436da76148a97acd1868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8015254d551147de8b1379c32c88ac8f"
I20250113 03:09:17.590538 27934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "48403f235703436da76148a97acd1868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d53b1c670a4095a7c570f6585f7c50"
I20250113 03:09:17.590610 28085 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:17.590624 27934 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:17.591166 28085 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48403f235703436da76148a97acd1868 in term 1.
I20250113 03:09:17.591238 27934 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48403f235703436da76148a97acd1868 in term 1.
I20250113 03:09:17.591301 27976 leader_election.cc:304] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48403f235703436da76148a97acd1868, 8015254d551147de8b1379c32c88ac8f; no voters:
I20250113 03:09:17.591416 28132 raft_consensus.cc:2798] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:17.591529 28132 raft_consensus.cc:695] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 LEADER]: Becoming Leader. State: Replica: 48403f235703436da76148a97acd1868, State: Running, Role: LEADER
I20250113 03:09:17.591579 28132 consensus_queue.cc:237] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:17.592023 27830 catalog_manager.cc:5526] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48403f235703436da76148a97acd1868 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "48403f235703436da76148a97acd1868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:17.602785 28085 raft_consensus.cc:1270] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 1 FOLLOWER]: Refusing update from remote peer 48403f235703436da76148a97acd1868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:17.602789 27934 raft_consensus.cc:1270] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 1 FOLLOWER]: Refusing update from remote peer 48403f235703436da76148a97acd1868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:17.602990 28136 consensus_queue.cc:1035] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:17.603107 28132 consensus_queue.cc:1035] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:28.027195 28188 raft_consensus.cc:988] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: : Instructing follower b0d53b1c670a4095a7c570f6585f7c50 to start an election
I20250113 03:09:28.027287 28188 raft_consensus.cc:1076] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 LEADER]: Signalling peer b0d53b1c670a4095a7c570f6585f7c50 to start an election
I20250113 03:09:28.027477 27934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3"
dest_uuid: "b0d53b1c670a4095a7c570f6585f7c50"
from {username='slave'} at 127.0.0.1:39144
I20250113 03:09:28.027542 27934 raft_consensus.cc:491] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:09:28.027561 27934 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:28.028155 27934 raft_consensus.cc:513] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:28.028314 27934 leader_election.cc:290] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [CANDIDATE]: Term 2 election: Requested vote from peers 48403f235703436da76148a97acd1868 (127.20.18.66:43507), 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193)
I20250113 03:09:28.030719 28010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "b0d53b1c670a4095a7c570f6585f7c50" candidate_term: 2 candidate_status { last_received { term: 1 index: 15980 } } ignore_live_leader: true dest_uuid: "48403f235703436da76148a97acd1868"
I20250113 03:09:28.030700 28084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f5be83c9ee4c8eba79ac1b661edaa3" candidate_uuid: "b0d53b1c670a4095a7c570f6585f7c50" candidate_term: 2 candidate_status { last_received { term: 1 index: 15980 } } ignore_live_leader: true dest_uuid: "8015254d551147de8b1379c32c88ac8f"
I20250113 03:09:28.030795 28010 raft_consensus.cc:3049] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:09:28.030810 28084 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:28.030818 28010 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48403f235703436da76148a97acd1868, State: Running, Role: LEADER
I20250113 03:09:28.030897 28010 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15980, Committed index: 15980, Last appended: 1.15980, Last appended by leader: 15980, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:28.030970 28010 raft_consensus.cc:3054] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:28.031404 28084 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d53b1c670a4095a7c570f6585f7c50 in term 2.
I20250113 03:09:28.031505 28010 raft_consensus.cc:2463] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d53b1c670a4095a7c570f6585f7c50 in term 2.
I20250113 03:09:28.031565 27900 leader_election.cc:304] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8015254d551147de8b1379c32c88ac8f, b0d53b1c670a4095a7c570f6585f7c50; no voters:
I20250113 03:09:28.031729 28195 raft_consensus.cc:2798] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:09:28.031837 28195 raft_consensus.cc:695] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 2 LEADER]: Becoming Leader. State: Replica: b0d53b1c670a4095a7c570f6585f7c50, State: Running, Role: LEADER
I20250113 03:09:28.031884 28195 consensus_queue.cc:237] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15980, Committed index: 15980, Last appended: 1.15980, Last appended by leader: 15980, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:28.032251 27831 catalog_manager.cc:5526] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 reported cstate change: term changed from 1 to 2, leader changed from 48403f235703436da76148a97acd1868 (127.20.18.66) to b0d53b1c670a4095a7c570f6585f7c50 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "b0d53b1c670a4095a7c570f6585f7c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:28.037498 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:09:28.039489 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:28.039561 20553 tablet_replica.cc:331] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: stopping tablet replica
I20250113 03:09:28.039600 20553 raft_consensus.cc:2238] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:28.039641 20553 raft_consensus.cc:2267] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:28.055802 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:09:28.056449 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:09:28.059171 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:28.059252 20553 tablet_replica.cc:331] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: stopping tablet replica
I20250113 03:09:28.059305 20553 raft_consensus.cc:2238] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:28.059350 20553 raft_consensus.cc:2267] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:28.072609 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
W20250113 03:09:28.078969 27899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.18.66:43507: connect: Connection refused (error 111)
W20250113 03:09:28.079299 27899 consensus_peers.cc:487] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 -> Peer 48403f235703436da76148a97acd1868 (127.20.18.66:43507): Couldn't send request to peer 48403f235703436da76148a97acd1868. Status: Network error: Client connection negotiation failed: client connection to 127.20.18.66:43507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 03:09:28.079299 27900 consensus_peers.cc:487] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 -> Peer 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193): Couldn't send request to peer 8015254d551147de8b1379c32c88ac8f. Status: Network error: Client connection negotiation failed: client connection to 127.20.18.67:37193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 03:09:29.077325 28202 meta_cache.cc:302] tablet 92f5be83c9ee4c8eba79ac1b661edaa3: replica b0d53b1c670a4095a7c570f6585f7c50 (127.20.18.65:39997) has failed: Timed out: ParticipateInTransaction RPC to 127.20.18.65:39997 timed out after 1.000s (SENT)
I20250113 03:09:29.077704 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:29.079149 28209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:29.079942 28210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:29.080744 20553 server_base.cc:1034] running on GCE node
W20250113 03:09:29.081301 28212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:29.081403 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:29.081430 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:29.081442 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737769081443 us; error 0 us; skew 500 ppm
I20250113 03:09:29.084900 20553 webserver.cc:458] Webserver started at http://127.20.18.66:39235/ using document root <none> and password file <none>
I20250113 03:09:29.084964 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:29.084990 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:09:29.085554 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:29.086050 28217 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:29.086164 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:29.086210 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "48403f235703436da76148a97acd1868"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:29.086266 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:29.109756 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:29.109915 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:29.110038 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:29.110275 28224 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:09:29.110790 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:09:29.110821 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:09:29.110850 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:09:29.111218 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:09:29.111251 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap starting.
I20250113 03:09:29.111244 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:29.114339 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:43507
I20250113 03:09:29.114547 28282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:43507 every 8 connection(s)
I20250113 03:09:29.126289 28283 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40849
I20250113 03:09:29.126336 28283 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:29.126413 28283 heartbeater.cc:510] Master 127.20.18.126:40849 requested a full tablet report, sending...
I20250113 03:09:29.126655 27831 ts_manager.cc:194] Re-registered known tserver with Master: 48403f235703436da76148a97acd1868 (127.20.18.66:43507)
I20250113 03:09:29.126956 27831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55796
I20250113 03:09:29.733877 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap replayed 1/4 log segments. Stats: ops{read=4623 overwritten=0 applied=4620 ignored=0} inserts{seen=230950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:09:30.127422 28283 heartbeater.cc:502] Master 127.20.18.126:40849 was elected leader, sending a full tablet report...
I20250113 03:09:30.398160 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap replayed 2/4 log segments. Stats: ops{read=9243 overwritten=0 applied=9240 ignored=0} inserts{seen=461950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20250113 03:09:30.506259 27900 consensus_peers.cc:487] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 -> Peer 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193): Couldn't send request to peer 8015254d551147de8b1379c32c88ac8f. Status: Network error: Client connection negotiation failed: client connection to 127.20.18.67:37193: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 03:09:30.661772 27899 consensus_peers.cc:487] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 -> Peer 48403f235703436da76148a97acd1868 (127.20.18.66:43507): Couldn't send request to peer 48403f235703436da76148a97acd1868. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250113 03:09:31.065761 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap replayed 3/4 log segments. Stats: ops{read=13864 overwritten=0 applied=13864 ignored=0} inserts{seen=693150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:31.379696 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap replayed 4/4 log segments. Stats: ops{read=15980 overwritten=0 applied=15980 ignored=0} inserts{seen=798950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:31.380105 28224 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Bootstrap complete.
I20250113 03:09:31.385244 28224 ts_tablet_manager.cc:1397] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Time spent bootstrapping tablet: real 2.274s user 1.983s sys 0.284s
I20250113 03:09:31.385387 28224 raft_consensus.cc:357] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:31.385448 28224 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48403f235703436da76148a97acd1868, State: Initialized, Role: FOLLOWER
I20250113 03:09:31.385499 28224 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15980, Last appended: 1.15980, Last appended by leader: 15980, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:31.385643 28224 ts_tablet_manager.cc:1428] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:31.385828 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:31.387583 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:31.388162 28297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:31.388417 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:31.389036 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:31.389115 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:31.389137 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:31.389148 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737771389148 us; error 0 us; skew 500 ppm
I20250113 03:09:31.389801 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43223/ using document root <none> and password file <none>
I20250113 03:09:31.389875 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:31.389899 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:09:31.390316 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:31.390667 28302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:31.390753 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:31.390794 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "8015254d551147de8b1379c32c88ac8f"
format_stamp: "Formatted at 2025-01-13 03:09:17 on dist-test-slave-d4dl"
I20250113 03:09:31.390833 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:31.410641 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:31.410777 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:31.410883 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:31.411149 28309 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:09:31.411653 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:09:31.411688 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:09:31.411718 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:09:31.412062 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:09:31.412088 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:31.412093 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap starting.
I20250113 03:09:31.415118 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:37193
I20250113 03:09:31.415174 28367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:37193 every 8 connection(s)
I20250113 03:09:31.417971 28368 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40849
I20250113 03:09:31.418030 28368 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:31.418123 28368 heartbeater.cc:510] Master 127.20.18.126:40849 requested a full tablet report, sending...
I20250113 03:09:31.418376 27831 ts_manager.cc:194] Re-registered known tserver with Master: 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193)
I20250113 03:09:31.418669 27831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46090
W20250113 03:09:31.630805 28378 rpcz_store.cc:267] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.0.1:33268 (ReqId={client: 6c7cb574f7594d01982c8f43c098b944, seq_no=0, attempt_no=0}) took 3552 ms (client timeout 999 ms). Trace:
W20250113 03:09:31.630946 28378 rpcz_store.cc:269] 0113 03:09:28.078605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 03:09:28.078627 (+ 22us) service_pool.cc:225] Handling call
0113 03:09:31.630769 (+3552142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 03:09:28.078684 (+ 0us) participant_op.cc:163] PREPARE: starting
0113 03:09:28.078687 (+ 3us) participant_op.cc:199] PREPARE: finished
0113 03:09:28.078692 (+ 5us) participant_op.cc:212] START. Timestamp: 1736737768078691
0113 03:09:28.078693 (+ 1us) op_driver.cc:348] REPLICATION: starting
0113 03:09:28.078715 (+ 22us) log.cc:844] Serialized 66 byte log entry
0113 03:09:31.630366 (+3551651us) op_driver.cc:464] REPLICATION: finished
0113 03:09:31.630700 (+ 334us) participant_op.cc:265] APPLY: starting
0113 03:09:31.630742 (+ 42us) participant_op.cc:270] APPLY: finished
0113 03:09:31.630750 (+ 8us) log.cc:844] Serialized 15 byte log entry
0113 03:09:31.630762 (+ 12us) participant_op.cc:94] Released txn lock
0113 03:09:31.630763 (+ 1us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":326,"prepare.queue_time_us":18,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":39,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"replication_time_us":3551670,"spinlock_wait_cycles":20096,"thread_start_us":47,"threads_started":1}]]}
I20250113 03:09:32.048640 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap replayed 1/4 log segments. Stats: ops{read=4931 overwritten=0 applied=4929 ignored=0} inserts{seen=246400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:09:32.419171 28368 heartbeater.cc:502] Master 127.20.18.126:40849 was elected leader, sending a full tablet report...
I20250113 03:09:32.759217 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap replayed 2/4 log segments. Stats: ops{read=9869 overwritten=0 applied=9866 ignored=0} inserts{seen=493250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20250113 03:09:33.019907 27900 consensus_peers.cc:487] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 -> Peer 8015254d551147de8b1379c32c88ac8f (127.20.18.67:37193): Couldn't send request to peer 8015254d551147de8b1379c32c88ac8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20250113 03:09:33.476954 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap replayed 3/4 log segments. Stats: ops{read=14810 overwritten=0 applied=14809 ignored=0} inserts{seen=740400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:09:33.655771 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap replayed 4/4 log segments. Stats: ops{read=15980 overwritten=0 applied=15980 ignored=0} inserts{seen=798950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:33.656165 28309 tablet_bootstrap.cc:492] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Bootstrap complete.
I20250113 03:09:33.661443 28309 ts_tablet_manager.cc:1397] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Time spent bootstrapping tablet: real 2.249s user 1.925s sys 0.324s
I20250113 03:09:33.661628 28309 raft_consensus.cc:357] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:33.661688 28309 raft_consensus.cc:738] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8015254d551147de8b1379c32c88ac8f, State: Initialized, Role: FOLLOWER
I20250113 03:09:33.661731 28309 consensus_queue.cc:260] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15980, Last appended: 1.15980, Last appended by leader: 15980, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48403f235703436da76148a97acd1868" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43507 } } peers { permanent_uuid: "b0d53b1c670a4095a7c570f6585f7c50" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39997 } } peers { permanent_uuid: "8015254d551147de8b1379c32c88ac8f" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 37193 } }
I20250113 03:09:33.661886 28309 ts_tablet_manager.cc:1428] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:34.174211 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:09:34.176604 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:34.176712 20553 tablet_replica.cc:331] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50: stopping tablet replica
I20250113 03:09:34.176761 20553 raft_consensus.cc:2238] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:09:34.176837 20553 raft_consensus.cc:2267] T 92f5be83c9ee4c8eba79ac1b661edaa3 P b0d53b1c670a4095a7c570f6585f7c50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:34.192198 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:09:34.193243 20553 tablet_server.cc:178] TabletServer@127.20.18.66:43507 shutting down...
I20250113 03:09:34.195094 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:34.195238 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:09:34.195276 20553 raft_consensus.cc:2238] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:34.195315 20553 raft_consensus.cc:2267] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 48403f235703436da76148a97acd1868 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:34.216467 20553 tablet_server.cc:195] TabletServer@127.20.18.66:43507 shutdown complete.
I20250113 03:09:34.217154 20553 tablet_server.cc:178] TabletServer@127.20.18.67:37193 shutting down...
I20250113 03:09:34.219319 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:34.219429 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:09:34.219484 20553 raft_consensus.cc:2238] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:34.219530 20553 raft_consensus.cc:2267] T 92f5be83c9ee4c8eba79ac1b661edaa3 P 8015254d551147de8b1379c32c88ac8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:34.232452 20553 tablet_server.cc:195] TabletServer@127.20.18.67:37193 shutdown complete.
I20250113 03:09:34.233233 20553 master.cc:537] Master@127.20.18.126:40849 shutting down...
I20250113 03:09:34.235291 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:09:34.235345 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:34.235364 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2dea1f47ce904d03bbdbe73b653e5b56: stopping tablet replica
I20250113 03:09:34.246683 20553 master.cc:559] Master@127.20.18.126:40849 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority (16862 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap
I20250113 03:09:34.266525 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:45429
I20250113 03:09:34.266690 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:34.268430 28389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.269080 28391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.269326 28388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:34.270087 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:34.270190 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:34.270222 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:34.270242 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737774270242 us; error 0 us; skew 500 ppm
I20250113 03:09:34.273988 20553 webserver.cc:458] Webserver started at http://127.20.18.126:35491/ using document root <none> and password file <none>
I20250113 03:09:34.274071 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:34.274117 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:34.274168 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:34.274423 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "d79f39649ec2419b8ec48cd3471a0941"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.275262 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.275692 28396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.275810 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.275852 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "d79f39649ec2419b8ec48cd3471a0941"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.275898 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:34.291265 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:34.291448 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:34.293838 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:45429
I20250113 03:09:34.304316 28448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:45429 every 8 connection(s)
I20250113 03:09:34.304437 28449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:34.305428 28449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941: Bootstrap starting.
I20250113 03:09:34.305756 28449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:34.306203 28449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941: No bootstrap required, opened a new log
I20250113 03:09:34.306318 28449 raft_consensus.cc:357] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER }
I20250113 03:09:34.306367 28449 raft_consensus.cc:383] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:34.306380 28449 raft_consensus.cc:738] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79f39649ec2419b8ec48cd3471a0941, State: Initialized, Role: FOLLOWER
I20250113 03:09:34.306433 28449 consensus_queue.cc:260] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER }
I20250113 03:09:34.306480 28449 raft_consensus.cc:397] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:09:34.306507 28449 raft_consensus.cc:491] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:09:34.306536 28449 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:34.306984 28449 raft_consensus.cc:513] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER }
I20250113 03:09:34.307035 28449 leader_election.cc:304] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d79f39649ec2419b8ec48cd3471a0941; no voters:
I20250113 03:09:34.307144 28449 leader_election.cc:290] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:09:34.307209 28452 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:34.307312 28452 raft_consensus.cc:695] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 1 LEADER]: Becoming Leader. State: Replica: d79f39649ec2419b8ec48cd3471a0941, State: Running, Role: LEADER
I20250113 03:09:34.307380 28452 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER }
I20250113 03:09:34.307307 28449 sys_catalog.cc:564] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:09:34.307618 28453 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d79f39649ec2419b8ec48cd3471a0941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER } }
I20250113 03:09:34.307696 28453 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:34.308010 28454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d79f39649ec2419b8ec48cd3471a0941. Latest consensus state: current_term: 1 leader_uuid: "d79f39649ec2419b8ec48cd3471a0941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79f39649ec2419b8ec48cd3471a0941" member_type: VOTER } }
I20250113 03:09:34.308081 28454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:34.308152 28456 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:09:34.308279 28456 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:09:34.308879 28456 catalog_manager.cc:1348] Generated new cluster ID: a788964b0859459fad7538ced63dfec7
I20250113 03:09:34.308907 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:09:34.308923 28456 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:09:34.313887 28456 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:09:34.314250 28456 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:09:34.329562 28456 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941: Generated new TSK 0
I20250113 03:09:34.329643 28456 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:09:34.340669 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:34.342021 28470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.342796 28471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.343631 28473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:34.344231 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:34.344326 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:34.344354 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:34.344367 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737774344367 us; error 0 us; skew 500 ppm
I20250113 03:09:34.345057 20553 webserver.cc:458] Webserver started at http://127.20.18.65:42201/ using document root <none> and password file <none>
I20250113 03:09:34.345134 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:34.345199 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:34.345253 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:34.345536 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "17a051713a4b456a87224d8d60431287"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.346356 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:34.346760 28478 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.346882 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.346922 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "17a051713a4b456a87224d8d60431287"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.346968 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:34.369380 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:34.369589 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:34.369730 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:34.369910 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:34.369935 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.369964 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:34.369989 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.372519 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:37405
I20250113 03:09:34.372591 28541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:37405 every 8 connection(s)
I20250113 03:09:34.372882 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:34.374250 28546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.374910 28549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.375555 28547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:34.375963 28542 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
I20250113 03:09:34.376019 28542 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:34.376114 28542 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:34.376282 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:34.376408 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:09:34.376425 28413 ts_manager.cc:194] Registered new tserver with Master: 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405)
W20250113 03:09:34.376461 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:34.376480 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737774376480 us; error 0 us; skew 500 ppm
I20250113 03:09:34.377090 28413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52998
I20250113 03:09:34.377211 20553 webserver.cc:458] Webserver started at http://127.20.18.66:44267/ using document root <none> and password file <none>
I20250113 03:09:34.377367 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:34.377430 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:34.377491 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:34.377771 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "10b2c06f143d405394498ef8b6ea4013"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.378615 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.379053 28554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.379173 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.379228 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "10b2c06f143d405394498ef8b6ea4013"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.379263 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:34.386646 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:34.386782 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:34.386888 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:34.387018 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:34.387039 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.387061 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:34.387077 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.389533 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:45567
I20250113 03:09:34.389581 28617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:45567 every 8 connection(s)
I20250113 03:09:34.389837 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:34.391506 28622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.391687 28621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:34.392421 28624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:34.392583 28618 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
I20250113 03:09:34.392640 28618 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:34.392760 28618 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:34.392977 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:34.392982 28413 ts_manager.cc:194] Registered new tserver with Master: 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567)
I20250113 03:09:34.393102 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:34.393151 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:34.393167 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737774393168 us; error 0 us; skew 500 ppm
I20250113 03:09:34.393440 28413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53008
I20250113 03:09:34.393842 20553 webserver.cc:458] Webserver started at http://127.20.18.67:34507/ using document root <none> and password file <none>
I20250113 03:09:34.393932 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:34.393975 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:34.394022 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:34.394259 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "44da7ca0ce434dad84899c386313f990"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.395017 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.395395 28629 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.395509 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.395546 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "44da7ca0ce434dad84899c386313f990"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:34.395588 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:34.401700 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:34.401876 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:34.402007 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:34.402184 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:34.402207 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.402236 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:34.402261 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.405180 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34387
I20250113 03:09:34.405217 28692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34387 every 8 connection(s)
I20250113 03:09:34.407795 28693 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
I20250113 03:09:34.407847 28693 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:34.407938 28693 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:34.408139 28413 ts_manager.cc:194] Registered new tserver with Master: 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:34.408456 28413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53022
I20250113 03:09:34.408663 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003323855s
I20250113 03:09:34.408982 20553 test_util.cc:274] Using random seed: -971205356
I20250113 03:09:34.412861 28413 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53030:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:09:34.412968 28413 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:09:34.417136 28582 tablet_service.cc:1467] Processing CreateTablet for tablet 699a6ef3bed94613a74cfd19eb14e573 (DEFAULT_TABLE table=test-workload [id=2ec31f9839094f91b5b20e76c61509a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:34.417255 28582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699a6ef3bed94613a74cfd19eb14e573. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:34.417300 28506 tablet_service.cc:1467] Processing CreateTablet for tablet 699a6ef3bed94613a74cfd19eb14e573 (DEFAULT_TABLE table=test-workload [id=2ec31f9839094f91b5b20e76c61509a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:34.417380 28506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699a6ef3bed94613a74cfd19eb14e573. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:34.417596 28657 tablet_service.cc:1467] Processing CreateTablet for tablet 699a6ef3bed94613a74cfd19eb14e573 (DEFAULT_TABLE table=test-workload [id=2ec31f9839094f91b5b20e76c61509a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:34.417681 28657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699a6ef3bed94613a74cfd19eb14e573. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:34.418411 28708 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap starting.
I20250113 03:09:34.418799 28709 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap starting.
I20250113 03:09:34.418826 28708 tablet_bootstrap.cc:654] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:34.419185 28709 tablet_bootstrap.cc:654] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:34.419384 28708 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: No bootstrap required, opened a new log
I20250113 03:09:34.419437 28708 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.419528 28710 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap starting.
I20250113 03:09:34.419598 28708 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.419672 28708 raft_consensus.cc:383] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:34.419696 28708 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a051713a4b456a87224d8d60431287, State: Initialized, Role: FOLLOWER
I20250113 03:09:34.419776 28708 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.419785 28709 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: No bootstrap required, opened a new log
I20250113 03:09:34.419867 28709 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.419958 28708 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:34.420038 28542 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:34.420014 28709 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.420069 28709 raft_consensus.cc:383] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:34.420089 28709 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b2c06f143d405394498ef8b6ea4013, State: Initialized, Role: FOLLOWER
I20250113 03:09:34.419934 28710 tablet_bootstrap.cc:654] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:34.420135 28709 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.420303 28709 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:34.420336 28618 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:34.420897 28710 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: No bootstrap required, opened a new log
I20250113 03:09:34.420938 28710 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:34.421078 28710 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.421136 28710 raft_consensus.cc:383] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:34.421152 28710 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44da7ca0ce434dad84899c386313f990, State: Initialized, Role: FOLLOWER
I20250113 03:09:34.421203 28710 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.421331 28710 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:34.421365 28693 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:34.574570 28714 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:09:34.574672 28714 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.574858 28714 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:34.577245 28667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990" is_pre_election: true
I20250113 03:09:34.577248 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a051713a4b456a87224d8d60431287" is_pre_election: true
I20250113 03:09:34.577344 28667 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 0.
I20250113 03:09:34.577364 28516 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 0.
I20250113 03:09:34.577510 28555 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b2c06f143d405394498ef8b6ea4013, 17a051713a4b456a87224d8d60431287; no voters:
I20250113 03:09:34.577579 28714 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:09:34.577625 28714 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:09:34.577652 28714 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:34.578244 28714 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.578333 28714 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 1 election: Requested vote from peers 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:34.578527 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a051713a4b456a87224d8d60431287"
I20250113 03:09:34.578572 28516 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:34.578545 28667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990"
I20250113 03:09:34.578604 28667 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:34.579089 28516 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 1.
I20250113 03:09:34.579195 28667 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 1.
I20250113 03:09:34.579258 28555 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b2c06f143d405394498ef8b6ea4013, 17a051713a4b456a87224d8d60431287; no voters:
I20250113 03:09:34.579347 28714 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:34.579480 28714 raft_consensus.cc:695] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 LEADER]: Becoming Leader. State: Replica: 10b2c06f143d405394498ef8b6ea4013, State: Running, Role: LEADER
I20250113 03:09:34.579526 28714 consensus_queue.cc:237] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:34.580072 28413 catalog_manager.cc:5526] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "10b2c06f143d405394498ef8b6ea4013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:34.601655 28667 raft_consensus.cc:1270] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 1 FOLLOWER]: Refusing update from remote peer 10b2c06f143d405394498ef8b6ea4013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:34.601681 28516 raft_consensus.cc:1270] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 1 FOLLOWER]: Refusing update from remote peer 10b2c06f143d405394498ef8b6ea4013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:34.601944 28714 consensus_queue.cc:1035] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:34.602041 28718 consensus_queue.cc:1035] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:45.103754 28714 raft_consensus.cc:988] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: : Instructing follower 17a051713a4b456a87224d8d60431287 to start an election
I20250113 03:09:45.103828 28714 raft_consensus.cc:1076] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 LEADER]: Signalling peer 17a051713a4b456a87224d8d60431287 to start an election
I20250113 03:09:45.103968 28515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573"
dest_uuid: "17a051713a4b456a87224d8d60431287"
from {username='slave'} at 127.0.0.1:47952
I20250113 03:09:45.104044 28515 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:09:45.104064 28515 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:45.104631 28515 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:45.104756 28515 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 2 election: Requested vote from peers 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:45.106894 28666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 2 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: true dest_uuid: "44da7ca0ce434dad84899c386313f990"
I20250113 03:09:45.106959 28666 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:45.106926 28592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 2 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: true dest_uuid: "10b2c06f143d405394498ef8b6ea4013"
I20250113 03:09:45.106986 28592 raft_consensus.cc:3049] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:09:45.107017 28592 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10b2c06f143d405394498ef8b6ea4013, State: Running, Role: LEADER
I20250113 03:09:45.107070 28592 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15528, Committed index: 15528, Last appended: 1.15528, Last appended by leader: 15528, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:45.107142 28592 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:09:45.107590 28666 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a051713a4b456a87224d8d60431287 in term 2.
I20250113 03:09:45.107722 28592 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a051713a4b456a87224d8d60431287 in term 2.
I20250113 03:09:45.107774 28479 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a051713a4b456a87224d8d60431287, 44da7ca0ce434dad84899c386313f990; no voters:
I20250113 03:09:45.107957 28765 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:09:45.108064 28765 raft_consensus.cc:695] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 LEADER]: Becoming Leader. State: Replica: 17a051713a4b456a87224d8d60431287, State: Running, Role: LEADER
I20250113 03:09:45.108114 28765 consensus_queue.cc:237] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15528, Committed index: 15528, Last appended: 1.15528, Last appended by leader: 15528, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:45.108515 28413 catalog_manager.cc:5526] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 reported cstate change: term changed from 1 to 2, leader changed from 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66) to 17a051713a4b456a87224d8d60431287 (127.20.18.65). New cstate: current_term: 2 leader_uuid: "17a051713a4b456a87224d8d60431287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:45.120939 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:09:45.123589 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:45.123680 20553 tablet_replica.cc:331] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: stopping tablet replica
I20250113 03:09:45.123745 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:09:45.123801 20553 pending_rounds.cc:62] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Trying to abort 1 pending ops.
I20250113 03:09:45.123819 20553 pending_rounds.cc:69] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Aborting op as it isn't in flight: id { term: 2 index: 15529 } timestamp: 7113677967803404288 op_type: NO_OP noop_request { }
I20250113 03:09:45.123853 20553 raft_consensus.cc:2883] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:09:45.123870 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250113 03:09:45.127655 28771 meta_cache.cc:302] tablet 699a6ef3bed94613a74cfd19eb14e573: replica 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405) has failed: Network error: Client connection negotiation failed: client connection to 127.20.18.65:37405: connect: Connection refused (error 111)
W20250113 03:09:45.135668 28582 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41242: Illegal state: replica 10b2c06f143d405394498ef8b6ea4013 is not leader of this config: current role FOLLOWER
I20250113 03:09:45.137032 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:09:45.137777 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:45.139555 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:45.140110 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:45.140880 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:45.141546 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:45.141625 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:45.141651 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:45.141666 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737785141667 us; error 0 us; skew 500 ppm
I20250113 03:09:45.142323 20553 webserver.cc:458] Webserver started at http://127.20.18.65:42201/ using document root <none> and password file <none>
I20250113 03:09:45.142398 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:45.142433 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
W20250113 03:09:45.142923 28657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38178: Illegal state: replica 44da7ca0ce434dad84899c386313f990 is not leader of this config: current role FOLLOWER
I20250113 03:09:45.142925 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:45.143430 28788 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:45.143548 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:45.143587 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "17a051713a4b456a87224d8d60431287"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:45.143633 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:45.157052 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:45.157186 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:45.157294 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:45.157508 28795 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:09:45.157994 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:09:45.158022 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:09:45.158048 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:09:45.158447 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:09:45.158476 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:45.158488 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap starting.
W20250113 03:09:45.159996 28582 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41242: Illegal state: replica 10b2c06f143d405394498ef8b6ea4013 is not leader of this config: current role FOLLOWER
I20250113 03:09:45.161358 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:37405
I20250113 03:09:45.167752 28853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:37405 every 8 connection(s)
I20250113 03:09:45.170379 28854 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
W20250113 03:09:45.170380 28657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38178: Illegal state: replica 44da7ca0ce434dad84899c386313f990 is not leader of this config: current role FOLLOWER
I20250113 03:09:45.170445 28854 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:45.170527 28854 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:45.170786 28413 ts_manager.cc:194] Re-registered known tserver with Master: 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405)
I20250113 03:09:45.171054 28413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57974
I20250113 03:09:45.802831 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap replayed 1/4 log segments. Stats: ops{read=4992 overwritten=0 applied=4992 ignored=0} inserts{seen=249550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:46.171469 28854 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:46.509574 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap replayed 2/4 log segments. Stats: ops{read=9989 overwritten=0 applied=9988 ignored=0} inserts{seen=499350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:09:46.791203 28860 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:09:46.791288 28860 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:46.791409 28860 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:46.791757 28666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 3 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990" is_pre_election: true
I20250113 03:09:46.791846 28666 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 2.
I20250113 03:09:46.792080 28555 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b2c06f143d405394498ef8b6ea4013, 44da7ca0ce434dad84899c386313f990; no voters:
I20250113 03:09:46.792168 28860 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 03:09:46.792204 28860 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:09:46.792227 28860 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:09:46.792902 28860 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:46.793015 28860 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 election: Requested vote from peers 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:46.793124 28666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 3 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990"
I20250113 03:09:46.793190 28666 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:09:46.793846 28666 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b2c06f143d405394498ef8b6ea4013 in term 3.
I20250113 03:09:46.793983 28555 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b2c06f143d405394498ef8b6ea4013, 44da7ca0ce434dad84899c386313f990; no voters:
I20250113 03:09:46.794059 28860 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 03:09:46.794180 28860 raft_consensus.cc:695] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 LEADER]: Becoming Leader. State: Replica: 10b2c06f143d405394498ef8b6ea4013, State: Running, Role: LEADER
I20250113 03:09:46.794229 28860 consensus_queue.cc:237] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15528, Committed index: 15528, Last appended: 1.15528, Last appended by leader: 15528, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:46.794657 28413 catalog_manager.cc:5526] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 reported cstate change: term changed from 2 to 3, leader changed from 17a051713a4b456a87224d8d60431287 (127.20.18.65) to 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66). New cstate: current_term: 3 leader_uuid: "10b2c06f143d405394498ef8b6ea4013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:46.794755 28828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 3 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: false dest_uuid: "17a051713a4b456a87224d8d60431287"
I20250113 03:09:46.794757 28827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "10b2c06f143d405394498ef8b6ea4013" candidate_term: 3 candidate_status { last_received { term: 1 index: 15528 } } ignore_live_leader: false dest_uuid: "17a051713a4b456a87224d8d60431287" is_pre_election: true
W20250113 03:09:46.794929 28555 leader_election.cc:343] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405): Illegal state: must be running to vote when last-logged opid is not known
W20250113 03:09:46.794994 28555 leader_election.cc:343] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405): Illegal state: must be running to vote when last-logged opid is not known
W20250113 03:09:47.265606 28555 consensus_peers.cc:487] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 -> Peer 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405): Couldn't send request to peer 17a051713a4b456a87224d8d60431287. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 03:09:47.288298 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap replayed 3/4 log segments. Stats: ops{read=14969 overwritten=0 applied=14968 ignored=0} inserts{seen=748350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:09:47.371248 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap replayed 4/4 log segments. Stats: ops{read=15529 overwritten=0 applied=15528 ignored=0} inserts{seen=776350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:09:47.371682 28795 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Bootstrap complete.
I20250113 03:09:47.376394 28795 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Time spent bootstrapping tablet: real 2.218s user 1.955s sys 0.255s
I20250113 03:09:47.376557 28795 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:47.376631 28795 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a051713a4b456a87224d8d60431287, State: Initialized, Role: FOLLOWER
I20250113 03:09:47.376675 28795 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15528, Last appended: 2.15529, Last appended by leader: 15529, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:47.376843 28795 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Time spent starting tablet: real 0.000s user 0.004s sys 0.000s
W20250113 03:09:47.389330 28814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53688: Illegal state: replica 17a051713a4b456a87224d8d60431287 is not leader of this config: current role FOLLOWER
I20250113 03:09:47.399781 28666 raft_consensus.cc:1270] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Refusing update from remote peer 10b2c06f143d405394498ef8b6ea4013: Log matching property violated. Preceding OpId in replica: term: 1 index: 15528. Preceding OpId from leader: term: 3 index: 15529. (index mismatch)
I20250113 03:09:47.399962 28860 consensus_queue.cc:1035] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15529, Last known committed idx: 15528, Time since last communication: 0.000s
I20250113 03:09:47.671727 28827 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:09:47.672623 28827 pending_rounds.cc:77] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Aborting all ops after (but not including) 15528
I20250113 03:09:47.672667 28827 pending_rounds.cc:99] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287: Aborting uncommitted NO_OP operation due to leader change: 2.15529
I20250113 03:09:47.672710 28827 raft_consensus.cc:2883] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 03:09:47.877056 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:09:47.879772 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:47.879889 20553 tablet_replica.cc:331] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: stopping tablet replica
I20250113 03:09:47.879936 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 LEADER]: Raft consensus shutting down.
I20250113 03:09:47.879987 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:47.884091 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:09:47.885027 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:47.885720 28774 meta_cache.cc:302] tablet 699a6ef3bed94613a74cfd19eb14e573: replica 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567) has failed: Network error: Client connection negotiation failed: client connection to 127.20.18.66:45567: connect: Connection refused (error 111)
W20250113 03:09:47.886790 28885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:47.887521 28886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:47.888182 28888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:47.888767 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:47.888850 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:47.888875 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:47.888895 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737787888895 us; error 0 us; skew 500 ppm
I20250113 03:09:47.889570 20553 webserver.cc:458] Webserver started at http://127.20.18.66:44267/ using document root <none> and password file <none>
I20250113 03:09:47.889645 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:47.889675 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:09:47.890162 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:47.890538 28894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:47.890648 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:47.890686 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "10b2c06f143d405394498ef8b6ea4013"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:47.890735 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 03:09:47.892048 28814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53688: Illegal state: replica 17a051713a4b456a87224d8d60431287 is not leader of this config: current role FOLLOWER
W20250113 03:09:47.897435 28657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38178: Illegal state: replica 44da7ca0ce434dad84899c386313f990 is not leader of this config: current role FOLLOWER
I20250113 03:09:47.910530 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:47.910671 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:47.910768 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:47.911000 28901 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:09:47.911533 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:09:47.911563 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:09:47.911618 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:09:47.911999 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:09:47.912026 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:47.912030 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap starting.
I20250113 03:09:47.914989 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:45567
W20250113 03:09:47.916754 28814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53688: Illegal state: replica 17a051713a4b456a87224d8d60431287 is not leader of this config: current role FOLLOWER
I20250113 03:09:47.921631 28959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:45567 every 8 connection(s)
I20250113 03:09:47.924126 28960 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
I20250113 03:09:47.924178 28960 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:47.924273 28960 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:47.924572 28412 ts_manager.cc:194] Re-registered known tserver with Master: 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567)
I20250113 03:09:47.924896 28412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57982
W20250113 03:09:47.927243 28657 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38178: Illegal state: replica 44da7ca0ce434dad84899c386313f990 is not leader of this config: current role FOLLOWER
I20250113 03:09:48.512387 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap replayed 1/4 log segments. Stats: ops{read=4621 overwritten=0 applied=4620 ignored=0} inserts{seen=230950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:09:48.925421 28960 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:49.179977 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap replayed 2/4 log segments. Stats: ops{read=9241 overwritten=0 applied=9239 ignored=0} inserts{seen=461900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:09:49.379385 28967 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10b2c06f143d405394498ef8b6ea4013)
I20250113 03:09:49.379467 28967 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:49.379649 28967 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:49.379890 28968 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 10b2c06f143d405394498ef8b6ea4013)
I20250113 03:09:49.379976 28968 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:49.380452 28968 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 17a051713a4b456a87224d8d60431287 (127.20.18.65:37405), 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567)
I20250113 03:09:49.382910 28666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990" is_pre_election: true
I20250113 03:09:49.382989 28666 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a051713a4b456a87224d8d60431287 in term 3.
I20250113 03:09:49.383129 28789 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a051713a4b456a87224d8d60431287, 44da7ca0ce434dad84899c386313f990; no voters:
I20250113 03:09:49.383222 28967 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250113 03:09:49.383266 28967 raft_consensus.cc:491] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 10b2c06f143d405394498ef8b6ea4013)
I20250113 03:09:49.383286 28967 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:09:49.383438 28827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "44da7ca0ce434dad84899c386313f990" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "17a051713a4b456a87224d8d60431287" is_pre_election: true
I20250113 03:09:49.383639 28934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "10b2c06f143d405394498ef8b6ea4013" is_pre_election: true
W20250113 03:09:49.383872 28792 leader_election.cc:343] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567): Illegal state: must be running to vote when last-logged opid is not known
I20250113 03:09:49.384107 28967 raft_consensus.cc:513] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:49.384248 28967 leader_election.cc:290] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 election: Requested vote from peers 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567), 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:49.384234 28827 raft_consensus.cc:2388] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44da7ca0ce434dad84899c386313f990 in current term 4: Already voted for candidate 17a051713a4b456a87224d8d60431287 in this term.
I20250113 03:09:49.384426 28934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "10b2c06f143d405394498ef8b6ea4013"
I20250113 03:09:49.384438 28666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "17a051713a4b456a87224d8d60431287" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "44da7ca0ce434dad84899c386313f990"
I20250113 03:09:49.384574 28666 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 3 FOLLOWER]: Advancing to term 4
W20250113 03:09:49.384609 28792 leader_election.cc:343] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 election: Tablet error from VoteRequest() call to peer 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567): Illegal state: must be running to vote when last-logged opid is not known
I20250113 03:09:49.385152 28666 raft_consensus.cc:2463] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a051713a4b456a87224d8d60431287 in term 4.
I20250113 03:09:49.385303 28789 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17a051713a4b456a87224d8d60431287, 44da7ca0ce434dad84899c386313f990; no voters: 10b2c06f143d405394498ef8b6ea4013
I20250113 03:09:49.385382 28967 raft_consensus.cc:2798] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 03:09:49.385491 28967 raft_consensus.cc:695] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 LEADER]: Becoming Leader. State: Replica: 17a051713a4b456a87224d8d60431287, State: Running, Role: LEADER
I20250113 03:09:49.385560 28967 consensus_queue.cc:237] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16124, Committed index: 16124, Last appended: 3.16124, Last appended by leader: 16124, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:49.386019 28412 catalog_manager.cc:5526] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 reported cstate change: term changed from 3 to 4, leader changed from 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66) to 17a051713a4b456a87224d8d60431287 (127.20.18.65). New cstate: current_term: 4 leader_uuid: "17a051713a4b456a87224d8d60431287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:49.386308 28934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a6ef3bed94613a74cfd19eb14e573" candidate_uuid: "44da7ca0ce434dad84899c386313f990" candidate_term: 4 candidate_status { last_received { term: 3 index: 16124 } } ignore_live_leader: false dest_uuid: "10b2c06f143d405394498ef8b6ea4013" is_pre_election: true
W20250113 03:09:49.386493 28633 leader_election.cc:343] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567): Illegal state: must be running to vote when last-logged opid is not known
I20250113 03:09:49.386543 28633 leader_election.cc:304] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44da7ca0ce434dad84899c386313f990; no voters: 10b2c06f143d405394498ef8b6ea4013, 17a051713a4b456a87224d8d60431287
I20250113 03:09:49.386641 28968 raft_consensus.cc:2743] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250113 03:09:49.848567 28792 consensus_peers.cc:487] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 -> Peer 10b2c06f143d405394498ef8b6ea4013 (127.20.18.66:45567): Couldn't send request to peer 10b2c06f143d405394498ef8b6ea4013. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 03:09:49.858389 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap replayed 3/4 log segments. Stats: ops{read=13862 overwritten=0 applied=13858 ignored=0} inserts{seen=692850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250113 03:09:49.861984 28666 raft_consensus.cc:1270] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Refusing update from remote peer 17a051713a4b456a87224d8d60431287: Log matching property violated. Preceding OpId in replica: term: 3 index: 16124. Preceding OpId from leader: term: 4 index: 16125. (index mismatch)
I20250113 03:09:49.862216 28977 consensus_queue.cc:1035] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16125, Last known committed idx: 16124, Time since last communication: 0.000s
I20250113 03:09:50.122217 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap replayed 4/4 log segments. Stats: ops{read=16124 overwritten=0 applied=16124 ignored=0} inserts{seen=776350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:50.122617 28901 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Bootstrap complete.
I20250113 03:09:50.127635 28901 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Time spent bootstrapping tablet: real 2.216s user 1.924s sys 0.287s
I20250113 03:09:50.127769 28901 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:50.127840 28901 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b2c06f143d405394498ef8b6ea4013, State: Initialized, Role: FOLLOWER
I20250113 03:09:50.127885 28901 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16124, Last appended: 3.16124, Last appended by leader: 16124, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:50.128036 28901 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
W20250113 03:09:50.178982 28924 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41274: Illegal state: replica 10b2c06f143d405394498ef8b6ea4013 is not leader of this config: current role FOLLOWER
I20250113 03:09:50.302873 28934 raft_consensus.cc:3054] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:09:50.312857 28991 mvcc.cc:204] Tried to move back new op lower bound from 7113677989076811776 to 7113677985323610112. Current Snapshot: MvccSnapshot[applied={T|T < 7113677989076811776}]
I20250113 03:09:50.628213 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
W20250113 03:09:50.628621 28789 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
W20250113 03:09:50.629176 28789 consensus_peers.cc:487] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 -> Peer 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387): Couldn't send request to peer 44da7ca0ce434dad84899c386313f990. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250113 03:09:50.631368 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:50.631476 20553 tablet_replica.cc:331] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: stopping tablet replica
I20250113 03:09:50.631523 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:50.631559 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:50.641239 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:09:50.642653 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:50.644343 28998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:50.645161 20553 server_base.cc:1034] running on GCE node
W20250113 03:09:50.645987 28999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:50.646397 29001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:50.646529 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:50.646561 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:50.646577 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737790646578 us; error 0 us; skew 500 ppm
I20250113 03:09:50.650516 20553 webserver.cc:458] Webserver started at http://127.20.18.67:34507/ using document root <none> and password file <none>
I20250113 03:09:50.650600 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:50.650636 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:09:50.651185 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:50.651618 29006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:50.651717 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:50.651759 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "44da7ca0ce434dad84899c386313f990"
format_stamp: "Formatted at 2025-01-13 03:09:34 on dist-test-slave-d4dl"
I20250113 03:09:50.651808 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:50.659423 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:50.659562 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:50.659679 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:50.659891 29013 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:09:50.660341 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:09:50.660375 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:09:50.660404 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:09:50.660789 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:09:50.660818 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:50.660822 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap starting.
I20250113 03:09:50.663834 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34387
I20250113 03:09:50.666599 29071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34387 every 8 connection(s)
I20250113 03:09:50.671188 29072 heartbeater.cc:346] Connected to a master server at 127.20.18.126:45429
I20250113 03:09:50.671237 29072 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:50.671334 29072 heartbeater.cc:510] Master 127.20.18.126:45429 requested a full tablet report, sending...
I20250113 03:09:50.671572 28413 ts_manager.cc:194] Re-registered known tserver with Master: 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387)
I20250113 03:09:50.671881 28413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57994
I20250113 03:09:51.346941 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap replayed 1/4 log segments. Stats: ops{read=4988 overwritten=0 applied=4988 ignored=0} inserts{seen=249350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:51.672421 29072 heartbeater.cc:502] Master 127.20.18.126:45429 was elected leader, sending a full tablet report...
I20250113 03:09:52.058260 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap replayed 2/4 log segments. Stats: ops{read=9989 overwritten=0 applied=9986 ignored=0} inserts{seen=499250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:09:52.783360 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap replayed 3/4 log segments. Stats: ops{read=15001 overwritten=0 applied=14997 ignored=0} inserts{seen=749800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
W20250113 03:09:52.900517 28789 consensus_peers.cc:487] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 -> Peer 44da7ca0ce434dad84899c386313f990 (127.20.18.67:34387): Couldn't send request to peer 44da7ca0ce434dad84899c386313f990. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250113 03:09:52.920114 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap replayed 4/4 log segments. Stats: ops{read=16647 overwritten=0 applied=16647 ignored=0} inserts{seen=776350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:09:52.920506 29013 tablet_bootstrap.cc:492] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Bootstrap complete.
I20250113 03:09:52.925074 29013 ts_tablet_manager.cc:1397] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Time spent bootstrapping tablet: real 2.264s user 1.919s sys 0.344s
I20250113 03:09:52.925206 29013 raft_consensus.cc:357] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:52.925264 29013 raft_consensus.cc:738] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44da7ca0ce434dad84899c386313f990, State: Initialized, Role: FOLLOWER
I20250113 03:09:52.925311 29013 consensus_queue.cc:260] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16647, Last appended: 4.16647, Last appended by leader: 16647, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a051713a4b456a87224d8d60431287" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 37405 } } peers { permanent_uuid: "10b2c06f143d405394498ef8b6ea4013" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 45567 } } peers { permanent_uuid: "44da7ca0ce434dad84899c386313f990" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34387 } }
I20250113 03:09:52.925468 29013 ts_tablet_manager.cc:1428] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:09:53.557893 20553 tablet_server.cc:178] TabletServer@127.20.18.65:37405 shutting down...
I20250113 03:09:53.560271 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:53.560372 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:09:53.560417 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 LEADER]: Raft consensus shutting down.
I20250113 03:09:53.560708 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 17a051713a4b456a87224d8d60431287 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:53.575793 20553 tablet_server.cc:195] TabletServer@127.20.18.65:37405 shutdown complete.
I20250113 03:09:53.576778 20553 tablet_server.cc:178] TabletServer@127.20.18.66:45567 shutting down...
I20250113 03:09:53.578905 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:53.579012 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:09:53.579058 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:53.579310 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 10b2c06f143d405394498ef8b6ea4013 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:53.599479 20553 tablet_server.cc:195] TabletServer@127.20.18.66:45567 shutdown complete.
I20250113 03:09:53.600412 20553 tablet_server.cc:178] TabletServer@127.20.18.67:34387 shutting down...
I20250113 03:09:53.602388 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:09:53.602494 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:09:53.602552 20553 raft_consensus.cc:2238] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250113 03:09:53.602703 20553 raft_consensus.cc:2267] T 699a6ef3bed94613a74cfd19eb14e573 P 44da7ca0ce434dad84899c386313f990 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:53.615592 20553 tablet_server.cc:195] TabletServer@127.20.18.67:34387 shutdown complete.
I20250113 03:09:53.616704 20553 master.cc:537] Master@127.20.18.126:45429 shutting down...
I20250113 03:09:53.618526 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:09:53.618575 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:09:53.618590 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P d79f39649ec2419b8ec48cd3471a0941: stopping tablet replica
I20250113 03:09:53.629781 20553 master.cc:559] Master@127.20.18.126:45429 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap (19381 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound
I20250113 03:09:53.647513 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:46427
I20250113 03:09:53.647652 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:53.649322 29097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.650110 29098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.650820 29100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:53.651486 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:53.651585 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:53.651610 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:53.651621 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737793651622 us; error 0 us; skew 500 ppm
I20250113 03:09:53.652158 20553 webserver.cc:458] Webserver started at http://127.20.18.126:43965/ using document root <none> and password file <none>
I20250113 03:09:53.652216 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:53.652261 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:53.652331 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:53.652679 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "bac8fe02495f4bb29e6f5f4c80b4e273"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.653450 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:53.653870 29105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.653973 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.654013 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "bac8fe02495f4bb29e6f5f4c80b4e273"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.654053 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:53.674779 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:53.674944 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:53.677364 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:46427
I20250113 03:09:53.677870 29158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:53.678716 29157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:46427 every 8 connection(s)
I20250113 03:09:53.681035 29158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273: Bootstrap starting.
I20250113 03:09:53.681341 29158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:53.681769 29158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273: No bootstrap required, opened a new log
I20250113 03:09:53.681879 29158 raft_consensus.cc:357] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER }
I20250113 03:09:53.681922 29158 raft_consensus.cc:383] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:53.681936 29158 raft_consensus.cc:738] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac8fe02495f4bb29e6f5f4c80b4e273, State: Initialized, Role: FOLLOWER
I20250113 03:09:53.681982 29158 consensus_queue.cc:260] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER }
I20250113 03:09:53.682029 29158 raft_consensus.cc:397] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:09:53.682050 29158 raft_consensus.cc:491] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:09:53.682080 29158 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:53.682523 29158 raft_consensus.cc:513] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER }
I20250113 03:09:53.682574 29158 leader_election.cc:304] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bac8fe02495f4bb29e6f5f4c80b4e273; no voters:
I20250113 03:09:53.682673 29158 leader_election.cc:290] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:09:53.682723 29161 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:53.682806 29158 sys_catalog.cc:564] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:09:53.682929 29161 raft_consensus.cc:695] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 1 LEADER]: Becoming Leader. State: Replica: bac8fe02495f4bb29e6f5f4c80b4e273, State: Running, Role: LEADER
I20250113 03:09:53.682981 29161 consensus_queue.cc:237] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER }
I20250113 03:09:53.683183 29162 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER } }
I20250113 03:09:53.683185 29163 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bac8fe02495f4bb29e6f5f4c80b4e273. Latest consensus state: current_term: 1 leader_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac8fe02495f4bb29e6f5f4c80b4e273" member_type: VOTER } }
I20250113 03:09:53.683290 29163 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:53.683303 29162 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [sys.catalog]: This master's current role is: LEADER
I20250113 03:09:53.683581 29168 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:09:53.683707 29168 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:09:53.684046 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:09:53.684201 29168 catalog_manager.cc:1348] Generated new cluster ID: 89b9ad4d27ef48298413b26d98419c1d
I20250113 03:09:53.684242 29168 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:09:53.697125 29168 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:09:53.697454 29168 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:09:53.700381 29168 catalog_manager.cc:5899] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273: Generated new TSK 0
I20250113 03:09:53.700469 29168 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:09:53.715668 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:53.717269 29179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.717851 29182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.718449 29180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:53.719290 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:53.719359 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:53.719396 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:53.719410 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737793719410 us; error 0 us; skew 500 ppm
I20250113 03:09:53.720031 20553 webserver.cc:458] Webserver started at http://127.20.18.65:42499/ using document root <none> and password file <none>
I20250113 03:09:53.720100 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:53.720137 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:53.720180 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:53.720386 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "55fabf616fa844b5a9922d2743a82acf"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.721161 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:53.721544 29187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.721653 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.721688 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "55fabf616fa844b5a9922d2743a82acf"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.721720 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:53.748093 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:53.748234 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:53.748338 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:53.748478 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:53.748499 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.748543 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:53.748567 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.751057 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:46197
I20250113 03:09:53.751513 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:09:53.751845 29250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:46197 every 8 connection(s)
W20250113 03:09:53.753139 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.753924 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.754463 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:53.754910 29251 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46427
I20250113 03:09:53.754931 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:53.755007 29251 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:53.755038 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:53.755067 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:53.755082 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737793755083 us; error 0 us; skew 500 ppm
I20250113 03:09:53.755112 29251 heartbeater.cc:510] Master 127.20.18.126:46427 requested a full tablet report, sending...
I20250113 03:09:53.755311 29122 ts_manager.cc:194] Registered new tserver with Master: 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197)
I20250113 03:09:53.755697 20553 webserver.cc:458] Webserver started at http://127.20.18.66:41421/ using document root <none> and password file <none>
I20250113 03:09:53.755769 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:53.755803 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:53.755839 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:53.756029 29122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50206
I20250113 03:09:53.756067 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "f47a943ad30a449392b3fe208cd1e792"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.756855 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:53.757182 29263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.757282 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.757310 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "f47a943ad30a449392b3fe208cd1e792"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.757331 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:53.761943 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:53.762058 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:53.762144 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:53.762260 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:53.762281 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.762297 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:53.762315 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.764571 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:37815
I20250113 03:09:53.764600 29326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:37815 every 8 connection(s)
I20250113 03:09:53.764840 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:09:53.766273 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:09:53.766831 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:53.767202 29327 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46427
I20250113 03:09:53.767246 29327 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:53.767308 29327 heartbeater.cc:510] Master 127.20.18.126:46427 requested a full tablet report, sending...
W20250113 03:09:53.767472 29333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:09:53.767472 29122 ts_manager.cc:194] Registered new tserver with Master: f47a943ad30a449392b3fe208cd1e792 (127.20.18.66:37815)
I20250113 03:09:53.767788 29122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50212
I20250113 03:09:53.767979 20553 server_base.cc:1034] running on GCE node
I20250113 03:09:53.768091 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:09:53.768121 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:09:53.768142 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737793768142 us; error 0 us; skew 500 ppm
I20250113 03:09:53.768714 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43517/ using document root <none> and password file <none>
I20250113 03:09:53.768770 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:09:53.768818 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:09:53.768843 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:09:53.769022 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "9955186ab96a4b2198c773fb2daa7725"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.769789 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:09:53.770129 29338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.770233 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.770280 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "9955186ab96a4b2198c773fb2daa7725"
format_stamp: "Formatted at 2025-01-13 03:09:53 on dist-test-slave-d4dl"
I20250113 03:09:53.770316 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:09:53.780831 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:09:53.780979 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:09:53.781086 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:09:53.781229 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:09:53.781250 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.781284 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:09:53.781306 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.783862 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:33613
I20250113 03:09:53.783903 29401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:33613 every 8 connection(s)
I20250113 03:09:53.786310 29402 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46427
I20250113 03:09:53.786361 29402 heartbeater.cc:463] Registering TS with master...
I20250113 03:09:53.786443 29402 heartbeater.cc:510] Master 127.20.18.126:46427 requested a full tablet report, sending...
I20250113 03:09:53.786629 29122 ts_manager.cc:194] Registered new tserver with Master: 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67:33613)
I20250113 03:09:53.786917 29122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50214
I20250113 03:09:53.787468 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003358072s
I20250113 03:09:53.787750 20553 test_util.cc:274] Using random seed: -951826587
I20250113 03:09:53.791606 29122 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50230:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:09:53.791698 29122 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:09:53.795513 29291 tablet_service.cc:1467] Processing CreateTablet for tablet b8891f1aa6864d3e93cb8ece5fd62e72 (DEFAULT_TABLE table=test-workload [id=ca26ee6f43fb48e0af9d60f512120037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:53.795626 29291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8891f1aa6864d3e93cb8ece5fd62e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:53.795646 29215 tablet_service.cc:1467] Processing CreateTablet for tablet b8891f1aa6864d3e93cb8ece5fd62e72 (DEFAULT_TABLE table=test-workload [id=ca26ee6f43fb48e0af9d60f512120037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:53.795737 29215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8891f1aa6864d3e93cb8ece5fd62e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:53.795738 29366 tablet_service.cc:1467] Processing CreateTablet for tablet b8891f1aa6864d3e93cb8ece5fd62e72 (DEFAULT_TABLE table=test-workload [id=ca26ee6f43fb48e0af9d60f512120037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:09:53.795863 29366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8891f1aa6864d3e93cb8ece5fd62e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:09:53.796847 29417 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: Bootstrap starting.
I20250113 03:09:53.797145 29418 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap starting.
I20250113 03:09:53.797272 29417 tablet_bootstrap.cc:654] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:53.797586 29418 tablet_bootstrap.cc:654] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:53.797791 29417 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: No bootstrap required, opened a new log
I20250113 03:09:53.797840 29417 ts_tablet_manager.cc:1397] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:53.797879 29419 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: Bootstrap starting.
I20250113 03:09:53.797967 29417 raft_consensus.cc:357] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.798033 29417 raft_consensus.cc:383] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:53.798061 29417 raft_consensus.cc:738] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f47a943ad30a449392b3fe208cd1e792, State: Initialized, Role: FOLLOWER
I20250113 03:09:53.798132 29417 consensus_queue.cc:260] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.798254 29419 tablet_bootstrap.cc:654] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: Neither blocks nor log segments found. Creating new log.
I20250113 03:09:53.798278 29417 ts_tablet_manager.cc:1428] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.798285 29327 heartbeater.cc:502] Master 127.20.18.126:46427 was elected leader, sending a full tablet report...
I20250113 03:09:53.798660 29418 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: No bootstrap required, opened a new log
I20250113 03:09:53.798722 29418 ts_tablet_manager.cc:1397] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250113 03:09:53.798857 29418 raft_consensus.cc:357] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.798908 29418 raft_consensus.cc:383] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:53.798928 29418 raft_consensus.cc:738] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fabf616fa844b5a9922d2743a82acf, State: Initialized, Role: FOLLOWER
I20250113 03:09:53.798990 29418 consensus_queue.cc:260] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.799127 29418 ts_tablet_manager.cc:1428] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.799154 29251 heartbeater.cc:502] Master 127.20.18.126:46427 was elected leader, sending a full tablet report...
I20250113 03:09:53.799314 29419 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: No bootstrap required, opened a new log
I20250113 03:09:53.799355 29419 ts_tablet_manager.cc:1397] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:09:53.799579 29419 raft_consensus.cc:357] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.799651 29419 raft_consensus.cc:383] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:09:53.799671 29419 raft_consensus.cc:738] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9955186ab96a4b2198c773fb2daa7725, State: Initialized, Role: FOLLOWER
I20250113 03:09:53.799726 29419 consensus_queue.cc:260] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.799896 29419 ts_tablet_manager.cc:1428] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:09:53.799930 29402 heartbeater.cc:502] Master 127.20.18.126:46427 was elected leader, sending a full tablet report...
I20250113 03:09:53.867511 29422 raft_consensus.cc:491] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:09:53.867561 29422 raft_consensus.cc:513] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.867691 29422 leader_election.cc:290] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197), 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67:33613)
I20250113 03:09:53.870429 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "f47a943ad30a449392b3fe208cd1e792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9955186ab96a4b2198c773fb2daa7725" is_pre_election: true
I20250113 03:09:53.870429 29225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "f47a943ad30a449392b3fe208cd1e792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fabf616fa844b5a9922d2743a82acf" is_pre_election: true
I20250113 03:09:53.870520 29225 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f47a943ad30a449392b3fe208cd1e792 in term 0.
I20250113 03:09:53.870519 29376 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f47a943ad30a449392b3fe208cd1e792 in term 0.
I20250113 03:09:53.870683 29265 leader_election.cc:304] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9955186ab96a4b2198c773fb2daa7725, f47a943ad30a449392b3fe208cd1e792; no voters:
I20250113 03:09:53.870779 29422 raft_consensus.cc:2798] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:09:53.870832 29422 raft_consensus.cc:491] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:09:53.870851 29422 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:53.871338 29422 raft_consensus.cc:513] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.871457 29422 leader_election.cc:290] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [CANDIDATE]: Term 1 election: Requested vote from peers 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197), 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67:33613)
I20250113 03:09:53.871575 29225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "f47a943ad30a449392b3fe208cd1e792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fabf616fa844b5a9922d2743a82acf"
I20250113 03:09:53.871593 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "f47a943ad30a449392b3fe208cd1e792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9955186ab96a4b2198c773fb2daa7725"
I20250113 03:09:53.871625 29225 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:53.871636 29376 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:09:53.872141 29225 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47a943ad30a449392b3fe208cd1e792 in term 1.
I20250113 03:09:53.872139 29376 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47a943ad30a449392b3fe208cd1e792 in term 1.
I20250113 03:09:53.872304 29264 leader_election.cc:304] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fabf616fa844b5a9922d2743a82acf, f47a943ad30a449392b3fe208cd1e792; no voters:
I20250113 03:09:53.872376 29422 raft_consensus.cc:2798] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:09:53.872470 29422 raft_consensus.cc:695] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 LEADER]: Becoming Leader. State: Replica: f47a943ad30a449392b3fe208cd1e792, State: Running, Role: LEADER
I20250113 03:09:53.872506 29422 consensus_queue.cc:237] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:09:53.872944 29122 catalog_manager.cc:5526] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 reported cstate change: term changed from 0 to 1, leader changed from <none> to f47a943ad30a449392b3fe208cd1e792 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "f47a943ad30a449392b3fe208cd1e792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20250113 03:09:53.893925 29376 raft_consensus.cc:1270] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 1 FOLLOWER]: Refusing update from remote peer f47a943ad30a449392b3fe208cd1e792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:53.893936 29225 raft_consensus.cc:1270] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 1 FOLLOWER]: Refusing update from remote peer f47a943ad30a449392b3fe208cd1e792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:09:53.894135 29427 consensus_queue.cc:1035] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:09:53.894213 29422 consensus_queue.cc:1035] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:04.409719 29427 raft_consensus.cc:988] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: : Instructing follower 55fabf616fa844b5a9922d2743a82acf to start an election
I20250113 03:10:04.409791 29427 raft_consensus.cc:1076] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 LEADER]: Signalling peer 55fabf616fa844b5a9922d2743a82acf to start an election
I20250113 03:10:04.409986 29224 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72"
dest_uuid: "55fabf616fa844b5a9922d2743a82acf"
from {username='slave'} at 127.0.0.1:50282
I20250113 03:10:04.410065 29224 raft_consensus.cc:491] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 03:10:04.410086 29224 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:04.410647 29224 raft_consensus.cc:513] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:04.410763 29224 leader_election.cc:290] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [CANDIDATE]: Term 2 election: Requested vote from peers f47a943ad30a449392b3fe208cd1e792 (127.20.18.66:37815), 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67:33613)
I20250113 03:10:04.413002 29376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "55fabf616fa844b5a9922d2743a82acf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: true dest_uuid: "9955186ab96a4b2198c773fb2daa7725"
I20250113 03:10:04.413045 29301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "55fabf616fa844b5a9922d2743a82acf" candidate_term: 2 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: true dest_uuid: "f47a943ad30a449392b3fe208cd1e792"
I20250113 03:10:04.413081 29376 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:04.413084 29301 raft_consensus.cc:3049] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:10:04.413132 29301 raft_consensus.cc:738] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f47a943ad30a449392b3fe208cd1e792, State: Running, Role: LEADER
I20250113 03:10:04.413195 29301 consensus_queue.cc:260] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16261, Committed index: 16261, Last appended: 1.16261, Last appended by leader: 16261, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:04.413270 29301 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:04.413743 29376 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fabf616fa844b5a9922d2743a82acf in term 2.
I20250113 03:10:04.413879 29301 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fabf616fa844b5a9922d2743a82acf in term 2.
I20250113 03:10:04.413940 29189 leader_election.cc:304] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fabf616fa844b5a9922d2743a82acf, 9955186ab96a4b2198c773fb2daa7725; no voters:
I20250113 03:10:04.414111 29486 raft_consensus.cc:2798] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:10:04.414212 29486 raft_consensus.cc:695] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 LEADER]: Becoming Leader. State: Replica: 55fabf616fa844b5a9922d2743a82acf, State: Running, Role: LEADER
I20250113 03:10:04.414263 29486 consensus_queue.cc:237] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16261, Committed index: 16261, Last appended: 1.16261, Last appended by leader: 16261, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:04.414629 29118 catalog_manager.cc:5526] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf reported cstate change: term changed from 1 to 2, leader changed from f47a943ad30a449392b3fe208cd1e792 (127.20.18.66) to 55fabf616fa844b5a9922d2743a82acf (127.20.18.65). New cstate: current_term: 2 leader_uuid: "55fabf616fa844b5a9922d2743a82acf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:04.427266 20553 tablet_replica.cc:331] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: stopping tablet replica
I20250113 03:10:04.427311 20553 raft_consensus.cc:2238] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:10:04.427364 20553 pending_rounds.cc:62] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Trying to abort 1 pending ops.
I20250113 03:10:04.427392 20553 pending_rounds.cc:69] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Aborting op as it isn't in flight: id { term: 2 index: 16262 } timestamp: 7113678046881226752 op_type: NO_OP noop_request { }
I20250113 03:10:04.427425 20553 raft_consensus.cc:2883] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:10:04.427459 20553 raft_consensus.cc:2267] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:04.432334 20553 ts_tablet_manager.cc:1905] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250113 03:10:04.433480 20553 ts_tablet_manager.cc:1918] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.16262
I20250113 03:10:04.433524 20553 log.cc:1198] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Deleting WAL directory at /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/b8891f1aa6864d3e93cb8ece5fd62e72
I20250113 03:10:05.974385 29502 raft_consensus.cc:491] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:05.974462 29502 raft_consensus.cc:513] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:05.974606 29502 leader_election.cc:290] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197), f47a943ad30a449392b3fe208cd1e792 (127.20.18.66:37815)
I20250113 03:10:05.977656 29224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "9955186ab96a4b2198c773fb2daa7725" candidate_term: 3 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: false dest_uuid: "55fabf616fa844b5a9922d2743a82acf" is_pre_election: true
I20250113 03:10:05.977727 29224 raft_consensus.cc:1798] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.16262
I20250113 03:10:05.977768 29224 raft_consensus.cc:2405] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9955186ab96a4b2198c773fb2daa7725 for term 3 because replica has last-logged OpId of term: 2 index: 16262, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 16261.
I20250113 03:10:05.977798 29301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "9955186ab96a4b2198c773fb2daa7725" candidate_term: 3 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: false dest_uuid: "f47a943ad30a449392b3fe208cd1e792" is_pre_election: true
I20250113 03:10:05.977852 29301 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9955186ab96a4b2198c773fb2daa7725 in term 2.
I20250113 03:10:05.977965 29340 leader_election.cc:304] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9955186ab96a4b2198c773fb2daa7725, f47a943ad30a449392b3fe208cd1e792; no voters: 55fabf616fa844b5a9922d2743a82acf
I20250113 03:10:05.978044 29502 raft_consensus.cc:2798] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 03:10:05.978078 29502 raft_consensus.cc:491] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:05.978094 29502 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:05.978650 29502 raft_consensus.cc:513] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:05.978745 29502 leader_election.cc:290] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [CANDIDATE]: Term 3 election: Requested vote from peers 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197), f47a943ad30a449392b3fe208cd1e792 (127.20.18.66:37815)
I20250113 03:10:05.978892 29224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "9955186ab96a4b2198c773fb2daa7725" candidate_term: 3 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: false dest_uuid: "55fabf616fa844b5a9922d2743a82acf"
I20250113 03:10:05.978905 29301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8891f1aa6864d3e93cb8ece5fd62e72" candidate_uuid: "9955186ab96a4b2198c773fb2daa7725" candidate_term: 3 candidate_status { last_received { term: 1 index: 16261 } } ignore_live_leader: false dest_uuid: "f47a943ad30a449392b3fe208cd1e792"
I20250113 03:10:05.978962 29224 raft_consensus.cc:1798] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.16262
I20250113 03:10:05.978971 29301 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:05.979002 29224 raft_consensus.cc:3054] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:05.979481 29301 raft_consensus.cc:2463] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9955186ab96a4b2198c773fb2daa7725 in term 3.
I20250113 03:10:05.979576 29224 raft_consensus.cc:2405] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9955186ab96a4b2198c773fb2daa7725 for term 3 because replica has last-logged OpId of term: 2 index: 16262, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 16261.
I20250113 03:10:05.979619 29340 leader_election.cc:304] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9955186ab96a4b2198c773fb2daa7725, f47a943ad30a449392b3fe208cd1e792; no voters:
I20250113 03:10:05.979701 29502 raft_consensus.cc:2798] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 3 FOLLOWER]: Leader election won for term 3
I20250113 03:10:05.979804 29502 raft_consensus.cc:695] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 3 LEADER]: Becoming Leader. State: Replica: 9955186ab96a4b2198c773fb2daa7725, State: Running, Role: LEADER
I20250113 03:10:05.979843 29502 consensus_queue.cc:237] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16261, Committed index: 16261, Last appended: 1.16261, Last appended by leader: 16261, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:05.980273 29118 catalog_manager.cc:5526] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 reported cstate change: term changed from 2 to 3, leader changed from 55fabf616fa844b5a9922d2743a82acf (127.20.18.65) to 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67). New cstate: current_term: 3 leader_uuid: "9955186ab96a4b2198c773fb2daa7725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:06.008953 29301 raft_consensus.cc:1270] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 3 FOLLOWER]: Refusing update from remote peer 9955186ab96a4b2198c773fb2daa7725: Log matching property violated. Preceding OpId in replica: term: 1 index: 16261. Preceding OpId from leader: term: 3 index: 16263. (index mismatch)
W20250113 03:10:06.009023 29339 consensus_peers.cc:487] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 -> Peer 55fabf616fa844b5a9922d2743a82acf (127.20.18.65:46197): Couldn't send request to peer 55fabf616fa844b5a9922d2743a82acf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250113 03:10:06.009111 29507 consensus_queue.cc:1035] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16262, Last known committed idx: 16261, Time since last communication: 0.000s
I20250113 03:10:06.562682 29523 ts_tablet_manager.cc:927] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Initiating tablet copy from peer 9955186ab96a4b2198c773fb2daa7725 (127.20.18.67:33613)
I20250113 03:10:06.562848 29523 tablet_copy_client.cc:323] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet copy: Beginning tablet copy session from remote peer at address 127.20.18.67:33613
I20250113 03:10:06.563102 29386 tablet_copy_service.cc:140] P 9955186ab96a4b2198c773fb2daa7725: Received BeginTabletCopySession request for tablet b8891f1aa6864d3e93cb8ece5fd62e72 from peer 55fabf616fa844b5a9922d2743a82acf ({username='slave'} at 127.0.0.1:57950)
I20250113 03:10:06.563161 29386 tablet_copy_service.cc:161] P 9955186ab96a4b2198c773fb2daa7725: Beginning new tablet copy session on tablet b8891f1aa6864d3e93cb8ece5fd62e72 from peer 55fabf616fa844b5a9922d2743a82acf at {username='slave'} at 127.0.0.1:57950: session id = 55fabf616fa844b5a9922d2743a82acf-b8891f1aa6864d3e93cb8ece5fd62e72
I20250113 03:10:06.563622 29386 tablet_copy_source_session.cc:215] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: Tablet Copy: opened 0 blocks and 4 log segments
I20250113 03:10:06.564013 29523 ts_tablet_manager.cc:1905] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250113 03:10:06.565474 29523 ts_tablet_manager.cc:1918] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.16262
I20250113 03:10:06.565557 29523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8891f1aa6864d3e93cb8ece5fd62e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:06.566021 29523 tablet_copy_client.cc:806] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet copy: Starting download of 0 data blocks...
I20250113 03:10:06.566116 29523 tablet_copy_client.cc:670] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet copy: Starting download of 4 WAL segments...
I20250113 03:10:06.598989 29523 tablet_copy_client.cc:538] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 03:10:06.599745 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap starting.
I20250113 03:10:07.220759 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap replayed 1/4 log segments. Stats: ops{read=4956 overwritten=0 applied=4955 ignored=0} inserts{seen=247700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:10:07.888105 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap replayed 2/4 log segments. Stats: ops{read=9896 overwritten=0 applied=9894 ignored=0} inserts{seen=494650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:08.564378 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap replayed 3/4 log segments. Stats: ops{read=14827 overwritten=0 applied=14824 ignored=0} inserts{seen=741150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:10:08.765883 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap replayed 4/4 log segments. Stats: ops{read=16385 overwritten=0 applied=16384 ignored=0} inserts{seen=813000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:10:08.766140 29523 tablet_bootstrap.cc:492] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Bootstrap complete.
I20250113 03:10:08.770826 29523 ts_tablet_manager.cc:1397] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Time spent bootstrapping tablet: real 2.171s user 1.864s sys 0.305s
I20250113 03:10:08.770994 29523 raft_consensus.cc:357] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:08.771164 29523 raft_consensus.cc:738] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fabf616fa844b5a9922d2743a82acf, State: Initialized, Role: FOLLOWER
I20250113 03:10:08.771214 29523 consensus_queue.cc:260] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16384, Last appended: 3.16385, Last appended by leader: 16385, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fabf616fa844b5a9922d2743a82acf" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 46197 } } peers { permanent_uuid: "f47a943ad30a449392b3fe208cd1e792" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 37815 } } peers { permanent_uuid: "9955186ab96a4b2198c773fb2daa7725" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 33613 } }
I20250113 03:10:08.771332 29523 ts_tablet_manager.cc:1428] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250113 03:10:08.771581 29385 tablet_copy_service.cc:342] P 9955186ab96a4b2198c773fb2daa7725: Request end of tablet copy session 55fabf616fa844b5a9922d2743a82acf-b8891f1aa6864d3e93cb8ece5fd62e72 received from {username='slave'} at 127.0.0.1:57950
I20250113 03:10:08.771642 29385 tablet_copy_service.cc:434] P 9955186ab96a4b2198c773fb2daa7725: ending tablet copy session 55fabf616fa844b5a9922d2743a82acf-b8891f1aa6864d3e93cb8ece5fd62e72 on tablet b8891f1aa6864d3e93cb8ece5fd62e72 with peer 55fabf616fa844b5a9922d2743a82acf
I20250113 03:10:08.831451 29224 raft_consensus.cc:1212] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.16261->[3.16262-3.16494] Dedup: 3.16385->[3.16386-3.16494]
I20250113 03:10:09.388731 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:10:09.390946 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:09.391031 20553 tablet_replica.cc:331] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf: stopping tablet replica
I20250113 03:10:09.391080 20553 raft_consensus.cc:2238] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:09.391119 20553 raft_consensus.cc:2267] T b8891f1aa6864d3e93cb8ece5fd62e72 P 55fabf616fa844b5a9922d2743a82acf [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:09.394661 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:10:09.395530 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:10:09.397629 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:09.397707 20553 tablet_replica.cc:331] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792: stopping tablet replica
I20250113 03:10:09.397753 20553 raft_consensus.cc:2238] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:09.397784 20553 raft_consensus.cc:2267] T b8891f1aa6864d3e93cb8ece5fd62e72 P f47a943ad30a449392b3fe208cd1e792 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:09.419194 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:10:09.419953 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:10:09.422178 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:09.422276 20553 tablet_replica.cc:331] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725: stopping tablet replica
I20250113 03:10:09.422314 20553 raft_consensus.cc:2238] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 3 LEADER]: Raft consensus shutting down.
I20250113 03:10:09.422374 20553 raft_consensus.cc:2267] T b8891f1aa6864d3e93cb8ece5fd62e72 P 9955186ab96a4b2198c773fb2daa7725 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:09.444288 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:10:09.445106 20553 master.cc:537] Master@127.20.18.126:46427 shutting down...
I20250113 03:10:09.447062 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:10:09.447116 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:09.447147 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P bac8fe02495f4bb29e6f5f4c80b4e273: stopping tablet replica
I20250113 03:10:09.458377 20553 master.cc:559] Master@127.20.18.126:46427 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound (15827 ms)
[----------] 21 tests from TxnParticipantITest (252752 ms total)
[----------] 2 tests from TxnParticipantElectionStormITest
[ RUN ] TxnParticipantElectionStormITest.TestFrequentElections
I20250113 03:10:09.475221 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:46381
I20250113 03:10:09.475356 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:09.476491 29556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.477169 29557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.477811 29559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.478691 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:09.478770 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:09.478798 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:09.478809 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737809478809 us; error 0 us; skew 500 ppm
I20250113 03:10:09.482440 20553 webserver.cc:458] Webserver started at http://127.20.18.126:39331/ using document root <none> and password file <none>
I20250113 03:10:09.482506 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:09.482547 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:09.482594 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:09.482808 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "7bec902c16a04168b317d0c3f9266f0f"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.483578 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:09.483958 29564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.484086 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.484126 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "7bec902c16a04168b317d0c3f9266f0f"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.484155 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:09.501379 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:09.501492 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:09.503410 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:46381
I20250113 03:10:09.504040 29617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:09.504715 29616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:46381 every 8 connection(s)
I20250113 03:10:09.506719 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Bootstrap starting.
I20250113 03:10:09.506932 29617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:09.507279 29617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: No bootstrap required, opened a new log
I20250113 03:10:09.507357 29617 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:09.507390 29617 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:09.507401 29617 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bec902c16a04168b317d0c3f9266f0f, State: Initialized, Role: FOLLOWER
I20250113 03:10:09.507470 29617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:09.507520 29620 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:09.507573 29620 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:09.507639 29620 leader_election.cc:304] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bec902c16a04168b317d0c3f9266f0f; no voters:
I20250113 03:10:09.507500 29617 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:10:09.507683 29617 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:10:09.507712 29617 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:09.507751 29620 leader_election.cc:290] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20250113 03:10:09.508145 29617 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:09.508212 29617 leader_election.cc:304] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bec902c16a04168b317d0c3f9266f0f; no voters:
I20250113 03:10:09.508246 29621 raft_consensus.cc:2758] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250113 03:10:09.508249 29617 leader_election.cc:290] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:10:09.508309 29620 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:09.508368 29617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:10:09.508417 29620 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 LEADER]: Becoming Leader. State: Replica: 7bec902c16a04168b317d0c3f9266f0f, State: Running, Role: LEADER
I20250113 03:10:09.508466 29620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:09.508616 29621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bec902c16a04168b317d0c3f9266f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER } }
I20250113 03:10:09.508674 29621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:09.508809 29625 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:10:09.508796 29622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bec902c16a04168b317d0c3f9266f0f. Latest consensus state: current_term: 1 leader_uuid: "7bec902c16a04168b317d0c3f9266f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER } }
I20250113 03:10:09.508863 29622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:09.509006 29625 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:10:09.509521 29625 catalog_manager.cc:1348] Generated new cluster ID: 10ac3db4a2cf46399e077d114d6bcc23
I20250113 03:10:09.509563 29625 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:10:09.509572 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:10:09.536921 29625 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:10:09.537261 29625 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:10:09.542135 29625 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Generated new TSK 0
I20250113 03:10:09.542194 29625 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:10:09.573310 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:09.574321 29638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.574891 29639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.575496 29641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.576295 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:09.576350 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:09.576376 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:09.576401 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737809576402 us; error 0 us; skew 500 ppm
I20250113 03:10:09.576874 20553 webserver.cc:458] Webserver started at http://127.20.18.65:37929/ using document root <none> and password file <none>
I20250113 03:10:09.576931 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:09.576972 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:09.577019 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:09.577222 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.577893 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:09.578240 29646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.578336 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.578372 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.578410 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:09.585878 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:09.586005 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:09.586117 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:09.586261 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:09.586285 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.586309 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:09.586331 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.588583 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:43087
I20250113 03:10:09.588737 29709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:43087 every 8 connection(s)
I20250113 03:10:09.588892 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:09.590060 29714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.590458 29715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:09.591096 29717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.591804 29710 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:09.591804 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:09.591871 29710 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:09.591909 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:09.591935 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:09.591959 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737809591959 us; error 0 us; skew 500 ppm
I20250113 03:10:09.591979 29710 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
I20250113 03:10:09.592173 29581 ts_manager.cc:194] Registered new tserver with Master: 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:09.592610 20553 webserver.cc:458] Webserver started at http://127.20.18.66:39569/ using document root <none> and password file <none>
I20250113 03:10:09.592674 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:09.592710 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:09.592751 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:09.592881 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56342
I20250113 03:10:09.592943 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.593621 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:09.593962 29722 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.594060 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.594095 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.594131 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:09.608831 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:09.608978 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:09.609083 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:09.609238 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:09.609266 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.609297 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:09.609318 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.611670 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:35093
I20250113 03:10:09.612075 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:10:09.613394 29785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:35093 every 8 connection(s)
W20250113 03:10:09.614111 29791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.614715 20553 server_base.cc:1034] running on GCE node
W20250113 03:10:09.615375 29789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.615806 29786 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:09.615862 29786 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:09.615942 29786 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
W20250113 03:10:09.615970 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:09.616134 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250113 03:10:09.616140 29581 ts_manager.cc:194] Registered new tserver with Master: 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093)
W20250113 03:10:09.616173 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:09.616195 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737809616196 us; error 0 us; skew 500 ppm
I20250113 03:10:09.616444 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56356
I20250113 03:10:09.616750 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43059/ using document root <none> and password file <none>
I20250113 03:10:09.616812 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:09.616858 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:09.616899 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:09.617113 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "773363024c05460c9bb9a4f875961edb"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.617823 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:09.618186 29797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.618289 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.618327 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "773363024c05460c9bb9a4f875961edb"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:09.618368 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:09.641574 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:09.641709 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:09.641820 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:09.641971 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:09.641993 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.642019 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:09.642045 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:09.644665 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:46757
I20250113 03:10:09.644716 29860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:46757 every 8 connection(s)
I20250113 03:10:09.646884 29861 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:09.646927 29861 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:09.647003 29861 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
I20250113 03:10:09.647181 29581 ts_manager.cc:194] Registered new tserver with Master: 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:09.647454 29581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56368
I20250113 03:10:09.648098 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003279999s
I20250113 03:10:09.648360 20553 test_util.cc:274] Using random seed: -935965976
I20250113 03:10:09.652040 29581 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56370:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:10:09.652148 29581 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:10:09.655819 29825 tablet_service.cc:1467] Processing CreateTablet for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9 (DEFAULT_TABLE table=test-workload [id=672a7a4ce6ac490eb281119f1e6fe6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:09.655915 29825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:09.656267 29674 tablet_service.cc:1467] Processing CreateTablet for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9 (DEFAULT_TABLE table=test-workload [id=672a7a4ce6ac490eb281119f1e6fe6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:09.656343 29674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:09.656847 29750 tablet_service.cc:1467] Processing CreateTablet for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9 (DEFAULT_TABLE table=test-workload [id=672a7a4ce6ac490eb281119f1e6fe6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:09.656932 29750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:09.657167 29876 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap starting.
I20250113 03:10:09.657322 29877 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap starting.
I20250113 03:10:09.657591 29876 tablet_bootstrap.cc:654] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:09.657711 29877 tablet_bootstrap.cc:654] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:09.658021 29876 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: No bootstrap required, opened a new log
I20250113 03:10:09.658061 29876 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:09.658216 29877 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: No bootstrap required, opened a new log
I20250113 03:10:09.658202 29876 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.658269 29877 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:09.658269 29876 raft_consensus.cc:383] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:09.658313 29876 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Initialized, Role: FOLLOWER
I20250113 03:10:09.658370 29876 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.658406 29877 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.658461 29877 raft_consensus.cc:383] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:09.658484 29877 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Initialized, Role: FOLLOWER
I20250113 03:10:09.658499 29876 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:09.658547 29877 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.658571 29861 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:09.658717 29877 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:09.658741 29710 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:09.658800 29880 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap starting.
I20250113 03:10:09.659224 29880 tablet_bootstrap.cc:654] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:09.659710 29880 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: No bootstrap required, opened a new log
I20250113 03:10:09.659754 29880 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:09.659865 29880 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.659902 29880 raft_consensus.cc:383] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:09.659924 29880 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Initialized, Role: FOLLOWER
I20250113 03:10:09.659968 29880 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.660131 29880 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:09.660180 29786 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:09.680809 29882 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:09.680858 29882 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.680989 29882 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:09.681087 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:09.681277 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.681731 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:09.683602 29881 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:09.683661 29881 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.683670 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" is_pre_election: true
I20250113 03:10:09.683740 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 0.
I20250113 03:10:09.683791 29881 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:09.684015 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb" is_pre_election: true
I20250113 03:10:09.684082 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 0.
I20250113 03:10:09.684197 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters:
I20250113 03:10:09.684278 29882 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:10:09.684320 29882 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:09.684350 29882 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:09.684875 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb" is_pre_election: true
I20250113 03:10:09.684952 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 0.
I20250113 03:10:09.684885 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" is_pre_election: true
I20250113 03:10:09.684964 29882 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.685062 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:09.685079 29882 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 1 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:09.685048 29684 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 1: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:09.685163 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:09.685150 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:10:09.685251 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:09.685238 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:09.685285 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:09.685794 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 1.
I20250113 03:10:09.685875 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 1.
I20250113 03:10:09.685918 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters:
I20250113 03:10:09.685933 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:09.685971 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:09.685976 29882 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:09.686084 29882 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:09.686129 29882 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.686501 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 1 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:09.686540 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.686635 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 2 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:09.686856 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:09.686905 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:09.687055 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:09.687409 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 2.
I20250113 03:10:09.687539 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:09.687671 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:10:09.687780 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:09.687830 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.688163 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" is_pre_election: true
I20250113 03:10:09.688169 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" is_pre_election: true
I20250113 03:10:09.688169 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 1 to 2, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 2 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:09.688386 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:09.688465 29881 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 03:10:09.698127 29684 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:10:09.698140 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 2 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 03:10:09.698184 29684 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:09.698230 29684 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:09.698285 29684 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:09.698352 29892 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:09.698897 29684 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 2 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250113 03:10:09.699049 29892 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:09.699108 29892 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer 52ecbe4b06b6444083ea4654c2c6cd33 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250113 03:10:09.699275 29684 pending_rounds.cc:77] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting all ops after (but not including) 0
I20250113 03:10:09.699307 29684 pending_rounds.cc:99] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250113 03:10:09.699329 29684 raft_consensus.cc:2883] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 03:10:19.722204 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:19.722270 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:19.722838 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.722970 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 3 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.723107 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 3 candidate_status { last_received { term: 2 index: 16839 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.723114 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 3 candidate_status { last_received { term: 2 index: 16839 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.723290 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:19.723377 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 03:10:19.727816 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 3 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 2. Current term is 3. Ops: []
I20250113 03:10:19.728003 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 2.16839, Next index: 16840, Last known committed idx: 16839, Time since last communication: 0.000s
W20250113 03:10:19.728106 29884 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250113 03:10:19.728133 29884 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250113 03:10:19.750262 29884 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 LEADER]: Stepping down as leader of term 2
I20250113 03:10:19.750303 29884 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:19.750357 29884 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16839, Committed index: 16839, Last appended: 2.16839, Last appended by leader: 16839, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.750420 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:19.754642 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:19.754730 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:10:19.755470 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.755599 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 4 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.755687 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 4 candidate_status { last_received { term: 2 index: 16839 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.755725 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 4 candidate_status { last_received { term: 2 index: 16839 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.755756 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:10:19.756244 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 4.
I20250113 03:10:19.756464 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 773363024c05460c9bb9a4f875961edb
I20250113 03:10:19.756568 29940 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 03:10:19.756726 29940 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:19.756773 29940 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16839, Committed index: 16839, Last appended: 2.16839, Last appended by leader: 16839, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.757154 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 2 to 4, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 4 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:19.757809 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 2 FOLLOWER]: Advancing to term 4
I20250113 03:10:19.757863 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 4 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 2 index: 16839. Preceding OpId from leader: term: 4 index: 16841. (index mismatch)
I20250113 03:10:19.758025 30046 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16840, Last known committed idx: 16839, Time since last communication: 0.000s
I20250113 03:10:19.758450 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 4 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 2 index: 16839. Preceding OpId from leader: term: 4 index: 16841. (index mismatch)
I20250113 03:10:19.758626 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16840, Last known committed idx: 16839, Time since last communication: 0.000s
I20250113 03:10:19.789040 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:19.789084 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 4 FOLLOWER]: Advancing to term 5
I20250113 03:10:19.789578 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.789691 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 5 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.789815 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.789829 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:19.789907 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:19.790050 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:19.790057 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 4.16842, Next index: 16843, Last known committed idx: 16841, Time since last communication: 0.000s
I20250113 03:10:19.790120 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 03:10:19.818549 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:19.818733 30046 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 4.16842, Next index: 16843, Last known committed idx: 16841, Time since last communication: 0.000s
I20250113 03:10:19.820817 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:19.820849 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 4 FOLLOWER]: Advancing to term 5
I20250113 03:10:19.821280 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.821377 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 5 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:19.821516 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 5 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.821566 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 5 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:19.821575 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 5: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:19.821751 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:19.821838 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 03:10:19.822988 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 5 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:19.823139 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 4.16842, Next index: 16843, Last known committed idx: 16842, Time since last communication: 0.000s
I20250113 03:10:19.825623 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:19.825672 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 5 FOLLOWER]: Advancing to term 6
I20250113 03:10:19.826054 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.826156 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 6 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.826244 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:19.826270 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 6 candidate_status { last_received { term: 4 index: 16842 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.826341 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 5 FOLLOWER]: Advancing to term 6
I20250113 03:10:19.826761 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 6.
I20250113 03:10:19.826900 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:19.826982 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 03:10:19.827031 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:19.827071 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16841, Committed index: 16841, Last appended: 4.16842, Last appended by leader: 16842, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.827402 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 4 to 6, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 6 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:19.829509 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 4. Current term is 6. Ops: [4.16843-4.16843]
I20250113 03:10:19.829587 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 6 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 4. Current term is 6. Ops: [4.16843-4.16843]
I20250113 03:10:19.829593 29683 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 03:10:19.829664 29683 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:19.829699 29683 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16842, Committed index: 16842, Last appended: 4.16843, Last appended by leader: 16843, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.829736 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 6 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 16842. Preceding OpId from leader: term: 6 index: 16844. (index mismatch)
I20250113 03:10:19.829751 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 4 FOLLOWER]: Advancing to term 6
I20250113 03:10:19.829905 29916 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16843, Last known committed idx: 16842, Time since last communication: 0.000s
I20250113 03:10:19.830440 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 6 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 16843. Preceding OpId from leader: term: 6 index: 16844. (index mismatch)
I20250113 03:10:19.830639 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16843, Last known committed idx: 16842, Time since last communication: 0.000s
I20250113 03:10:19.830708 29884 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer 52ecbe4b06b6444083ea4654c2c6cd33 log is divergent from this leader: its last log entry 4.16843 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16842
I20250113 03:10:19.830979 29683 pending_rounds.cc:77] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting all ops after (but not including) 16842
I20250113 03:10:19.831012 29683 pending_rounds.cc:99] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 4.16843
I20250113 03:10:19.842995 29899 mvcc.cc:204] Tried to move back new op lower bound from 7113678110021328896 to 7113678110012071936. Current Snapshot: MvccSnapshot[applied={T|T < 7113678110021328896 or (T in {7113678110021328896})}]
I20250113 03:10:19.874878 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:19.874922 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 6 FOLLOWER]: Advancing to term 7
I20250113 03:10:19.875360 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.875463 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 7 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.875567 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 7 candidate_status { last_received { term: 6 index: 16845 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.875581 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 7 candidate_status { last_received { term: 6 index: 16845 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.875761 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:19.875836 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 03:10:19.877573 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 6. Current term is 7. Ops: []
I20250113 03:10:19.877738 29916 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 6.16845, Next index: 16846, Last known committed idx: 16845, Time since last communication: 0.000s
I20250113 03:10:19.896061 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 6. Current term is 7. Ops: [6.16846-6.16846]
I20250113 03:10:19.896247 29916 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 6.16845, Next index: 16846, Last known committed idx: 16845, Time since last communication: 0.000s
I20250113 03:10:19.910405 29929 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 LEADER]: Stepping down as leader of term 6
I20250113 03:10:19.910445 29929 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:19.910483 29929 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16846, Committed index: 16846, Last appended: 6.16846, Last appended by leader: 16846, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.910558 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 6 FOLLOWER]: Advancing to term 7
I20250113 03:10:19.914463 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:19.914506 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 7 FOLLOWER]: Advancing to term 8
I20250113 03:10:19.914914 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.915019 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 8 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.915117 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 8 candidate_status { last_received { term: 6 index: 16845 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.915119 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 8 candidate_status { last_received { term: 6 index: 16845 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.915192 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 7 FOLLOWER]: Advancing to term 8
I20250113 03:10:19.915757 29760 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for term 8 because replica has last-logged OpId of term: 6 index: 16846, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 16845.
I20250113 03:10:19.915922 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:19.916015 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250113 03:10:19.927539 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:19.927584 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 6 FOLLOWER]: Advancing to term 7
I20250113 03:10:19.927973 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.928057 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 7 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:19.928189 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 7 candidate_status { last_received { term: 6 index: 16846 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:19.928205 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 7 candidate_status { last_received { term: 6 index: 16846 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:19.928246 29760 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 7. Current term is 8.
I20250113 03:10:19.928261 29683 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 7. Current term is 8.
I20250113 03:10:19.928349 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 7 election: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 7. Current term is 8.
I20250113 03:10:19.928375 29801 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250113 03:10:19.928431 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 7 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 7. Current term is 8.
I20250113 03:10:19.928449 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 7 FOLLOWER]: Advancing to term 8
I20250113 03:10:19.928834 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 7. Current term is 8.
I20250113 03:10:19.941776 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:19.941817 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 8 FOLLOWER]: Advancing to term 9
I20250113 03:10:19.942191 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.942291 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 9 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:19.942397 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 16846 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:19.942441 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 9 candidate_status { last_received { term: 6 index: 16846 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:19.942459 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 8 FOLLOWER]: Advancing to term 9
I20250113 03:10:19.942478 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 8 FOLLOWER]: Advancing to term 9
I20250113 03:10:19.942941 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 9.
I20250113 03:10:19.942941 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 9.
I20250113 03:10:19.943104 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:19.943217 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 FOLLOWER]: Leader election won for term 9
I20250113 03:10:19.943253 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:19.943291 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16845, Committed index: 16845, Last appended: 6.16846, Last appended by leader: 16846, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:19.943563 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20250113 03:10:19.952176 29929 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 9 is higher.
I20250113 03:10:19.959604 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 9 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 16845. Preceding OpId from leader: term: 9 index: 16847. (index mismatch)
I20250113 03:10:19.959606 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 9 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 6 index: 16846. Preceding OpId from leader: term: 9 index: 16847. (index mismatch)
I20250113 03:10:19.959813 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16847, Last known committed idx: 16845, Time since last communication: 0.000s
I20250113 03:10:19.959913 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16847, Last known committed idx: 16846, Time since last communication: 0.000s
I20250113 03:10:20.016721 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.016784 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 9 FOLLOWER]: Advancing to term 10
I20250113 03:10:20.017256 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.017352 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 10 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.017467 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 10 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.017467 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 10 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.017643 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.017717 30046 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250113 03:10:20.019640 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 9. Current term is 10. Ops: []
I20250113 03:10:20.019793 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 9.16847, Next index: 16848, Last known committed idx: 16847, Time since last communication: 0.000s
I20250113 03:10:20.047998 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.048051 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 9 FOLLOWER]: Advancing to term 10
I20250113 03:10:20.048210 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 9. Current term is 10. Ops: []
I20250113 03:10:20.048377 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 9.16847, Next index: 16848, Last known committed idx: 16847, Time since last communication: 0.000s
I20250113 03:10:20.048547 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.048664 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 10 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.048796 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 10 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.048856 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 10: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:20.048844 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 10 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.049027 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.049106 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250113 03:10:20.052160 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 10 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 9. Current term is 10. Ops: []
I20250113 03:10:20.052342 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 9.16847, Next index: 16848, Last known committed idx: 16847, Time since last communication: 0.000s
I20250113 03:10:20.053828 29935 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 LEADER]: Stepping down as leader of term 9
I20250113 03:10:20.053862 29935 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.053897 29935 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16847, Committed index: 16847, Last appended: 9.16847, Last appended by leader: 16847, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.053961 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 9 FOLLOWER]: Advancing to term 10
I20250113 03:10:20.058142 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.058189 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 10 FOLLOWER]: Advancing to term 11
I20250113 03:10:20.058547 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.058634 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 11 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.058787 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 11 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.058794 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 11 candidate_status { last_received { term: 9 index: 16847 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.058845 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 10 FOLLOWER]: Advancing to term 11
I20250113 03:10:20.058856 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 10 FOLLOWER]: Advancing to term 11
I20250113 03:10:20.059243 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 11.
I20250113 03:10:20.059257 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 11.
I20250113 03:10:20.059382 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:20.059446 30046 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 FOLLOWER]: Leader election won for term 11
I20250113 03:10:20.059490 30046 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.059525 30046 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16847, Committed index: 16847, Last appended: 9.16847, Last appended by leader: 16847, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.059836 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 9 to 11, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 11 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.084295 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 11 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 9 index: 16847. Preceding OpId from leader: term: 11 index: 16848. (index mismatch)
I20250113 03:10:20.084299 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 11 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 9 index: 16847. Preceding OpId from leader: term: 11 index: 16848. (index mismatch)
I20250113 03:10:20.084460 30046 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16848, Last known committed idx: 16847, Time since last communication: 0.000s
I20250113 03:10:20.084519 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16848, Last known committed idx: 16847, Time since last communication: 0.000s
I20250113 03:10:20.115357 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:20.115398 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 11 FOLLOWER]: Advancing to term 12
I20250113 03:10:20.115820 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.115926 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 12 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.116027 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:20.116031 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.116073 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 11 FOLLOWER]: Advancing to term 12
I20250113 03:10:20.116032 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 12 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.116403 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 12. Ops: []
I20250113 03:10:20.116560 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.116669 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.116770 29834 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 12: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:20.116784 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 12 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.116899 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 12 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.116885 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 12 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.116895 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.116971 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 12: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:20.116997 29935 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250113 03:10:20.117079 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.117146 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250113 03:10:20.122018 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 12. Ops: []
I20250113 03:10:20.122201 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.144161 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 12. Ops: []
I20250113 03:10:20.144196 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 12. Ops: []
I20250113 03:10:20.144362 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.144431 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.152424 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.152470 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 12 FOLLOWER]: Advancing to term 13
I20250113 03:10:20.152845 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.152941 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 13 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.153076 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.153103 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 13 candidate_status { last_received { term: 11 index: 16848 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.153162 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 12 FOLLOWER]: Advancing to term 13
I20250113 03:10:20.153559 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 13.
I20250113 03:10:20.153702 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:20.153781 29935 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 FOLLOWER]: Leader election won for term 13
I20250113 03:10:20.153832 29935 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.153873 29935 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16848, Committed index: 16848, Last appended: 11.16848, Last appended by leader: 16848, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.154198 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 11 to 13, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 13 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.177544 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 13. Ops: []
I20250113 03:10:20.177716 30046 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.177778 30046 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer 8fa8ecf2b7614f1490b642f1b5eaee9a log is divergent from this leader: its last log entry 13.16849 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16848
I20250113 03:10:20.178591 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 13 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 11. Current term is 13. Ops: []
I20250113 03:10:20.178741 30046 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 11.16848, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.180301 29683 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 LEADER]: Stepping down as leader of term 11
I20250113 03:10:20.180334 29683 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.180382 29683 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16848, Committed index: 16848, Last appended: 11.16848, Last appended by leader: 16848, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.180439 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 11 FOLLOWER]: Advancing to term 13
I20250113 03:10:20.180851 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 13 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 16848. Preceding OpId from leader: term: 13 index: 16849. (index mismatch)
I20250113 03:10:20.181033 29935 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.189051 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 13 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 11 index: 16848. Preceding OpId from leader: term: 13 index: 16849. (index mismatch)
I20250113 03:10:20.189210 29935 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16849, Last known committed idx: 16848, Time since last communication: 0.000s
I20250113 03:10:20.212584 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.212625 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 13 FOLLOWER]: Advancing to term 14
I20250113 03:10:20.213078 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.213177 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 14 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.213305 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 14 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.213315 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 14 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.213507 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.213546 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 14 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 13. Current term is 14. Ops: []
I20250113 03:10:20.213596 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250113 03:10:20.213743 29916 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 13.16849, Next index: 16850, Last known committed idx: 16849, Time since last communication: 0.000s
I20250113 03:10:20.241905 29884 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 LEADER]: Stepping down as leader of term 13
I20250113 03:10:20.241941 29884 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.241976 29884 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16849, Committed index: 16849, Last appended: 13.16849, Last appended by leader: 16849, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.242035 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 13 FOLLOWER]: Advancing to term 14
I20250113 03:10:20.251977 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.252030 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 14 FOLLOWER]: Advancing to term 15
I20250113 03:10:20.252416 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.252513 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 15 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.252628 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 15 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.252631 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 15 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.252693 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 14 FOLLOWER]: Advancing to term 15
I20250113 03:10:20.253111 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 15.
I20250113 03:10:20.253247 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.253324 30069 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 FOLLOWER]: Leader election won for term 15
I20250113 03:10:20.253366 30069 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.253409 30069 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16849, Committed index: 16849, Last appended: 13.16849, Last appended by leader: 16849, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.253720 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 13 to 15, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 15 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.269258 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 13 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.269301 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 13 FOLLOWER]: Advancing to term 14
I20250113 03:10:20.269644 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.269737 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 14 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.269853 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 14 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.269897 29760 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 14. Current term is 15.
I20250113 03:10:20.269881 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 14 candidate_status { last_received { term: 13 index: 16849 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.270044 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 14 election: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 14. Current term is 15.
I20250113 03:10:20.270076 29801 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20250113 03:10:20.270120 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 14 FOLLOWER]: Advancing to term 15
I20250113 03:10:20.270506 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 14. Current term is 15.
I20250113 03:10:20.276786 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 13 index: 16849. Preceding OpId from leader: term: 15 index: 16850. (index mismatch)
I20250113 03:10:20.276954 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16850, Last known committed idx: 16849, Time since last communication: 0.000s
I20250113 03:10:20.277812 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 15 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 13 index: 16849. Preceding OpId from leader: term: 15 index: 16850. (index mismatch)
I20250113 03:10:20.277963 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16850, Last known committed idx: 16849, Time since last communication: 0.000s
W20250113 03:10:20.320178 29940 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 15 is higher.
I20250113 03:10:20.332262 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:20.332305 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 15 FOLLOWER]: Advancing to term 16
I20250113 03:10:20.332684 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.332772 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 16 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.332913 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 16 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.332913 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 16 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.333093 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.333165 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250113 03:10:20.337224 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 16. Ops: []
I20250113 03:10:20.337388 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
W20250113 03:10:20.345317 29940 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 15 is higher.
W20250113 03:10:20.353449 29940 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 15 is higher.
I20250113 03:10:20.361752 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 16. Ops: []
I20250113 03:10:20.361925 30046 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.367061 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.367108 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 16 FOLLOWER]: Advancing to term 17
I20250113 03:10:20.367475 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.367564 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 17 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.367734 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 17 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.367734 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 17 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.367941 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.368026 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20250113 03:10:20.389566 29940 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 15 is higher.
I20250113 03:10:20.396404 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 17 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 17. Ops: []
I20250113 03:10:20.396591 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.427095 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.427143 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 17 FOLLOWER]: Advancing to term 18
I20250113 03:10:20.427485 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.427567 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 18 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.427723 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 18 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.427734 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 18 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.427903 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.427981 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20250113 03:10:20.428031 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 18. Ops: []
I20250113 03:10:20.428206 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.445672 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:20.445720 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 15 FOLLOWER]: Advancing to term 16
I20250113 03:10:20.446110 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.446204 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 16 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.446327 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.446333 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 16 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.446395 29834 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 16. Current term is 18.
I20250113 03:10:20.446470 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 16 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 16. Ops: []
I20250113 03:10:20.446497 29725 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 16 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 16. Current term is 18.
I20250113 03:10:20.446522 29725 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20250113 03:10:20.446563 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 16 FOLLOWER]: Advancing to term 18
I20250113 03:10:20.446650 30046 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.446985 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 18 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 16. Current term is 18.
I20250113 03:10:20.450450 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 15. Current term is 18. Ops: []
I20250113 03:10:20.450613 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 15.16850, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.451301 30069 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 LEADER]: Stepping down as leader of term 15
I20250113 03:10:20.451331 30069 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.451371 30069 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16850, Committed index: 16850, Last appended: 15.16850, Last appended by leader: 16850, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.451428 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 15 FOLLOWER]: Advancing to term 16
I20250113 03:10:20.475591 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.475638 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 16 FOLLOWER]: Advancing to term 17
I20250113 03:10:20.476013 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.476106 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 17 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.476223 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 17 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.476222 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 17 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.476284 29834 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 17. Current term is 18.
I20250113 03:10:20.476300 29760 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 17. Current term is 18.
I20250113 03:10:20.476400 29649 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 17 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 17. Current term is 18.
I20250113 03:10:20.476434 29649 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20250113 03:10:20.476454 29650 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 17 election: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 17. Current term is 18.
I20250113 03:10:20.476487 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 17 FOLLOWER]: Advancing to term 18
I20250113 03:10:20.476888 30046 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 17. Current term is 18.
I20250113 03:10:20.484347 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.484390 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 18 FOLLOWER]: Advancing to term 19
I20250113 03:10:20.484791 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.484881 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 19 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.484994 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 19 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.485001 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 19 candidate_status { last_received { term: 15 index: 16850 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.485071 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 18 FOLLOWER]: Advancing to term 19
I20250113 03:10:20.485075 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 18 FOLLOWER]: Advancing to term 19
I20250113 03:10:20.485580 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 19.
I20250113 03:10:20.485580 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 19.
I20250113 03:10:20.485735 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:20.485816 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 FOLLOWER]: Leader election won for term 19
I20250113 03:10:20.485867 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.485908 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16850, Committed index: 16850, Last appended: 15.16850, Last appended by leader: 16850, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.486215 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 15 to 19, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 19 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.508564 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 19 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 16850. Preceding OpId from leader: term: 19 index: 16851. (index mismatch)
I20250113 03:10:20.508739 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.517563 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 19 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 15 index: 16850. Preceding OpId from leader: term: 19 index: 16851. (index mismatch)
I20250113 03:10:20.517711 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16851, Last known committed idx: 16850, Time since last communication: 0.000s
I20250113 03:10:20.572217 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 19 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.572263 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 19 FOLLOWER]: Advancing to term 20
I20250113 03:10:20.572746 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.572840 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 20 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.572914 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 19. Current term is 20. Ops: []
I20250113 03:10:20.572932 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 20 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.572984 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 20 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.573127 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 19.16851, Next index: 16852, Last known committed idx: 16851, Time since last communication: 0.000s
I20250113 03:10:20.573148 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.573217 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20250113 03:10:20.594049 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.594094 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 19 FOLLOWER]: Advancing to term 20
I20250113 03:10:20.594468 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.594574 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 20 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.594692 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 20 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.594720 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 20 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.594803 29834 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 20: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:20.594897 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 20 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 19. Current term is 20. Ops: []
I20250113 03:10:20.594946 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.595052 30046 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20250113 03:10:20.595029 29916 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 19.16851, Next index: 16852, Last known committed idx: 16851, Time since last communication: 0.000s
I20250113 03:10:20.599306 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 19. Current term is 20. Ops: []
I20250113 03:10:20.599460 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 19.16851, Next index: 16852, Last known committed idx: 16851, Time since last communication: 0.000s
I20250113 03:10:20.609524 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.609565 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 20 FOLLOWER]: Advancing to term 21
I20250113 03:10:20.609937 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.610051 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 21 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.610162 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 21 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.610174 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 21 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.610227 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 20 FOLLOWER]: Advancing to term 21
I20250113 03:10:20.610656 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 21.
I20250113 03:10:20.610781 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters: 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.610857 30095 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 FOLLOWER]: Leader election won for term 21
I20250113 03:10:20.610905 30095 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:20.610949 30095 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16851, Committed index: 16851, Last appended: 19.16851, Last appended by leader: 16851, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.611268 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 19 to 21, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 21 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:20.613595 29929 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 LEADER]: Stepping down as leader of term 19
I20250113 03:10:20.613628 29929 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.613657 29929 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16851, Committed index: 16851, Last appended: 19.16851, Last appended by leader: 16851, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.613708 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 19 FOLLOWER]: Advancing to term 20
I20250113 03:10:20.633423 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 21 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 19 index: 16851. Preceding OpId from leader: term: 21 index: 16852. (index mismatch)
I20250113 03:10:20.633574 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16852, Last known committed idx: 16851, Time since last communication: 0.000s
I20250113 03:10:20.636933 29916 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.636974 29916 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 20 FOLLOWER]: Advancing to term 21
I20250113 03:10:20.637377 29916 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.637468 29916 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 21 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.637567 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.637601 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16851 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.637784 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.637853 29916 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20250113 03:10:20.640545 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 21 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 19 index: 16851. Preceding OpId from leader: term: 21 index: 16852. (index mismatch)
I20250113 03:10:20.640717 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16852, Last known committed idx: 16851, Time since last communication: 0.000s
I20250113 03:10:20.664685 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:20.664738 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 21 FOLLOWER]: Advancing to term 22
I20250113 03:10:20.665151 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.665252 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 22 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.665371 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 22 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.665403 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 22 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.665558 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.665643 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250113 03:10:20.665750 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 21. Current term is 22. Ops: []
I20250113 03:10:20.665916 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 21.16852, Next index: 16853, Last known committed idx: 16852, Time since last communication: 0.000s
I20250113 03:10:20.671165 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:20.671212 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 21 FOLLOWER]: Advancing to term 22
I20250113 03:10:20.671619 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.671712 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 22 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.671856 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.671856 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.671943 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 22: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:20.672058 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.672132 29929 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250113 03:10:20.677858 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 22 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 21. Current term is 22. Ops: []
I20250113 03:10:20.678017 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 21.16852, Next index: 16853, Last known committed idx: 16852, Time since last communication: 0.000s
I20250113 03:10:20.683653 30067 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 LEADER]: Stepping down as leader of term 21
I20250113 03:10:20.683689 30067 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:20.683730 30067 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16852, Committed index: 16852, Last appended: 21.16852, Last appended by leader: 16852, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.683794 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 21 FOLLOWER]: Advancing to term 22
I20250113 03:10:20.700325 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.700366 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 22 FOLLOWER]: Advancing to term 23
I20250113 03:10:20.700840 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.700939 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 23 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.701113 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 23 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.701148 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 23 candidate_status { last_received { term: 21 index: 16852 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.701187 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 22 FOLLOWER]: Advancing to term 23
I20250113 03:10:20.701186 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 22 FOLLOWER]: Advancing to term 23
I20250113 03:10:20.701723 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 23.
I20250113 03:10:20.701730 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 23.
I20250113 03:10:20.701886 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:20.701959 30069 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 FOLLOWER]: Leader election won for term 23
I20250113 03:10:20.702005 30069 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.702039 30069 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16852, Committed index: 16852, Last appended: 21.16852, Last appended by leader: 16852, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.702350 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 21 to 23, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 23 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.730741 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 23 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 21 index: 16852. Preceding OpId from leader: term: 23 index: 16853. (index mismatch)
I20250113 03:10:20.730954 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16853, Last known committed idx: 16852, Time since last communication: 0.000s
I20250113 03:10:20.732746 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 23 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 21 index: 16852. Preceding OpId from leader: term: 23 index: 16853. (index mismatch)
I20250113 03:10:20.732914 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16853, Last known committed idx: 16852, Time since last communication: 0.000s
I20250113 03:10:20.762578 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:20.762622 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 23 FOLLOWER]: Advancing to term 24
I20250113 03:10:20.763008 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.763091 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 24 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.763233 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16853 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.763249 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16853 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.763301 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 23 FOLLOWER]: Advancing to term 24
I20250113 03:10:20.763736 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 24.
I20250113 03:10:20.763851 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:20.763921 30067 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 FOLLOWER]: Leader election won for term 24
I20250113 03:10:20.763971 30067 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:20.764011 30067 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16853, Committed index: 16853, Last appended: 23.16853, Last appended by leader: 16853, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.764336 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 23 to 24, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 24 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:20.767344 29834 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 23. Current term is 24. Ops: []
I20250113 03:10:20.767520 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 23.16853, Next index: 16854, Last known committed idx: 16853, Time since last communication: 0.000s
I20250113 03:10:20.767572 30069 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer 773363024c05460c9bb9a4f875961edb log is divergent from this leader: its last log entry 24.16854 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16853
I20250113 03:10:20.769323 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 24 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 23. Current term is 24. Ops: []
I20250113 03:10:20.769487 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 23.16853, Next index: 16854, Last known committed idx: 16853, Time since last communication: 0.000s
I20250113 03:10:20.786394 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 24 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16853. Preceding OpId from leader: term: 24 index: 16854. (index mismatch)
I20250113 03:10:20.786569 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16854, Last known committed idx: 16853, Time since last communication: 0.000s
I20250113 03:10:20.792558 29683 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 LEADER]: Stepping down as leader of term 23
I20250113 03:10:20.792596 29683 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.792630 29683 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16853, Committed index: 16853, Last appended: 23.16853, Last appended by leader: 16853, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.792690 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 23 FOLLOWER]: Advancing to term 24
I20250113 03:10:20.793303 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 24 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16853. Preceding OpId from leader: term: 24 index: 16854. (index mismatch)
I20250113 03:10:20.793462 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16854, Last known committed idx: 16853, Time since last communication: 0.000s
I20250113 03:10:20.817860 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:20.817906 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 24 FOLLOWER]: Advancing to term 25
I20250113 03:10:20.818308 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.818401 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 25 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.818540 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.818552 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.818728 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.818797 29929 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250113 03:10:20.819813 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 24. Current term is 25. Ops: []
I20250113 03:10:20.819964 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 24.16854, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.849479 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 24. Current term is 25. Ops: []
I20250113 03:10:20.849645 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 24.16854, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.850304 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:20.850342 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 24 FOLLOWER]: Advancing to term 25
I20250113 03:10:20.850759 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.850863 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 25 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.850983 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 25 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.850992 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 25 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.851053 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 25: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:20.851204 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.851276 30046 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250113 03:10:20.851915 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 25 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 24. Current term is 25. Ops: []
I20250113 03:10:20.852075 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 24.16854, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.861735 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.861778 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 25 FOLLOWER]: Advancing to term 26
I20250113 03:10:20.862130 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.862216 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 26 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.862357 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.862357 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16854 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.862450 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 25 FOLLOWER]: Advancing to term 26
I20250113 03:10:20.862848 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 26.
I20250113 03:10:20.862986 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 773363024c05460c9bb9a4f875961edb
I20250113 03:10:20.863051 29929 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 FOLLOWER]: Leader election won for term 26
I20250113 03:10:20.863092 29929 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.863133 29929 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16854, Committed index: 16854, Last appended: 24.16854, Last appended by leader: 16854, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.863433 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 24 to 26, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 26 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:20.880414 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 LEADER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 24. Current term is 26. Ops: []
I20250113 03:10:20.880640 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 24.16854, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.880697 30067 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer 8fa8ecf2b7614f1490b642f1b5eaee9a log is divergent from this leader: its last log entry 26.16855 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16854
I20250113 03:10:20.882098 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 26 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 24. Current term is 26. Ops: []
I20250113 03:10:20.882266 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 24.16854, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.885473 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 26 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16854. Preceding OpId from leader: term: 26 index: 16855. (index mismatch)
I20250113 03:10:20.885648 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.895731 29834 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 LEADER]: Stepping down as leader of term 24
I20250113 03:10:20.895767 29834 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:20.895799 29834 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16854, Committed index: 16854, Last appended: 24.16854, Last appended by leader: 16854, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.895869 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 24 FOLLOWER]: Advancing to term 26
I20250113 03:10:20.896281 29834 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 26 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16854. Preceding OpId from leader: term: 26 index: 16855. (index mismatch)
I20250113 03:10:20.896450 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16855, Last known committed idx: 16854, Time since last communication: 0.000s
I20250113 03:10:20.916492 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.916551 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 26 FOLLOWER]: Advancing to term 27
I20250113 03:10:20.916965 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.917065 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 27 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.917204 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 27 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.917204 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 27 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.917410 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.917485 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250113 03:10:20.917802 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 26. Current term is 27. Ops: []
I20250113 03:10:20.917934 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 26.16855, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:20.941391 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 26. Current term is 27. Ops: []
I20250113 03:10:20.941551 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 26.16855, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:20.951429 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 26 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:20.951481 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 26 FOLLOWER]: Advancing to term 27
I20250113 03:10:20.951876 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.951954 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 27 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:20.952109 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.952131 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:20.952191 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 27: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:20.952301 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.952380 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250113 03:10:20.954830 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:20.954866 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 27 FOLLOWER]: Advancing to term 28
I20250113 03:10:20.955250 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.955345 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 28 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:20.955456 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 28 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:20.955456 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 28 candidate_status { last_received { term: 26 index: 16855 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:20.955539 29834 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 27 FOLLOWER]: Advancing to term 28
I20250113 03:10:20.955991 29834 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 28.
I20250113 03:10:20.956053 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 28 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 26. Current term is 28. Ops: []
I20250113 03:10:20.956143 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters: 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:20.956192 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 26.16855, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:20.956218 29940 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 FOLLOWER]: Leader election won for term 28
I20250113 03:10:20.956257 29940 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:20.956296 29940 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16855, Committed index: 16855, Last appended: 26.16855, Last appended by leader: 16855, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.956624 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 26 to 28, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 28 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20250113 03:10:20.959930 29899 lock_manager.cc:494] Txn 7 has not acquired the partition lock after 1000ms
I20250113 03:10:20.967922 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 LEADER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 26. Current term is 28. Ops: []
I20250113 03:10:20.968094 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 26.16855, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:20.968156 29935 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer 52ecbe4b06b6444083ea4654c2c6cd33 log is divergent from this leader: its last log entry 28.16856 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16855
I20250113 03:10:20.982549 29760 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 LEADER]: Stepping down as leader of term 26
I20250113 03:10:20.982585 29760 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:20.982617 29760 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16855, Committed index: 16855, Last appended: 26.16855, Last appended by leader: 16855, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:20.982685 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 26 FOLLOWER]: Advancing to term 28
I20250113 03:10:20.983090 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 28 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 26 index: 16855. Preceding OpId from leader: term: 28 index: 16856. (index mismatch)
I20250113 03:10:20.983271 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:20.992842 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 28 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 26 index: 16855. Preceding OpId from leader: term: 28 index: 16856. (index mismatch)
I20250113 03:10:20.993011 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16856, Last known committed idx: 16855, Time since last communication: 0.000s
I20250113 03:10:21.042222 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:21.042270 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 28 FOLLOWER]: Advancing to term 29
I20250113 03:10:21.042677 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.042794 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 29 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.042906 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.042922 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 29 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.043079 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:21.043159 29935 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20250113 03:10:21.044306 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 28. Current term is 29. Ops: []
I20250113 03:10:21.044472 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 28.16856, Next index: 16857, Last known committed idx: 16856, Time since last communication: 0.000s
I20250113 03:10:21.051303 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:21.051349 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 28 FOLLOWER]: Advancing to term 29
I20250113 03:10:21.051753 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.051838 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 29 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.051975 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.051982 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.052057 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 29: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:21.052168 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.052234 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20250113 03:10:21.054257 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 29 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 28. Current term is 29. Ops: []
I20250113 03:10:21.054399 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 28.16856, Next index: 16857, Last known committed idx: 16856, Time since last communication: 0.000s
I20250113 03:10:21.071902 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 28. Current term is 29. Ops: []
I20250113 03:10:21.072095 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 28.16856, Next index: 16857, Last known committed idx: 16856, Time since last communication: 0.000s
I20250113 03:10:21.080406 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.080466 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 29 FOLLOWER]: Advancing to term 30
I20250113 03:10:21.080854 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.080955 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 30 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.081054 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.081084 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 30 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.081120 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 29 FOLLOWER]: Advancing to term 30
I20250113 03:10:21.081322 30069 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 LEADER]: Stepping down as leader of term 28
I20250113 03:10:21.081360 30069 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:21.081398 30069 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16856, Committed index: 16856, Last appended: 28.16856, Last appended by leader: 16856, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.081466 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 28 FOLLOWER]: Advancing to term 29
I20250113 03:10:21.081629 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 30.
I20250113 03:10:21.081786 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:21.081861 29929 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 FOLLOWER]: Leader election won for term 30
I20250113 03:10:21.081916 29929 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.081960 29929 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16856, Committed index: 16856, Last appended: 28.16856, Last appended by leader: 16856, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.082264 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 28 to 30, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 30 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:21.109548 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 30 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 16856. Preceding OpId from leader: term: 30 index: 16857. (index mismatch)
I20250113 03:10:21.109742 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16857, Last known committed idx: 16856, Time since last communication: 0.000s
I20250113 03:10:21.111711 30046 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.111749 30046 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 29 FOLLOWER]: Advancing to term 30
I20250113 03:10:21.112120 30046 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.112224 30046 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 30 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.112321 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 30 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.112335 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 30 candidate_status { last_received { term: 28 index: 16856 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.112505 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 30 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 28 index: 16856. Preceding OpId from leader: term: 30 index: 16857. (index mismatch)
I20250113 03:10:21.112540 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.112627 30046 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20250113 03:10:21.112668 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16857, Last known committed idx: 16856, Time since last communication: 0.000s
I20250113 03:10:21.141239 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.141289 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 30 FOLLOWER]: Advancing to term 31
I20250113 03:10:21.141680 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.141758 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 31 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.141901 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 31 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.141906 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 31 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.142092 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.142164 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20250113 03:10:21.143913 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.143942 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 30 FOLLOWER]: Advancing to term 31
I20250113 03:10:21.144323 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.144415 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 31 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.144574 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 31 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.144557 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 31 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.144644 29835 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 31: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:21.145287 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 30. Current term is 31. Ops: []
I20250113 03:10:21.145453 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 30.16857, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.145658 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.145728 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20250113 03:10:21.148463 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 31 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 30. Current term is 31. Ops: []
I20250113 03:10:21.148623 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 30.16857, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.173705 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.173749 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 31 FOLLOWER]: Advancing to term 32
I20250113 03:10:21.174125 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.174214 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 32 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.174353 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 32 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.174358 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 32 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.174409 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 31 FOLLOWER]: Advancing to term 32
I20250113 03:10:21.174896 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 32.
I20250113 03:10:21.175050 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters: 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.175119 30067 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 FOLLOWER]: Leader election won for term 32
I20250113 03:10:21.175172 30067 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.175211 30067 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16857, Committed index: 16857, Last appended: 30.16857, Last appended by leader: 16857, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.175493 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 30 to 32, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 32 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:21.177663 29935 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 LEADER]: Stepping down as leader of term 30
I20250113 03:10:21.177709 29935 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.177759 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 LEADER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 30. Current term is 32. Ops: []
I20250113 03:10:21.177773 29935 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16857, Committed index: 16857, Last appended: 30.16857, Last appended by leader: 16857, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.177855 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 30 FOLLOWER]: Advancing to term 31
I20250113 03:10:21.206290 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.206331 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 31 FOLLOWER]: Advancing to term 32
I20250113 03:10:21.206811 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.206907 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 32 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.206915 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 32 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 30 index: 16857. Preceding OpId from leader: term: 32 index: 16858. (index mismatch)
I20250113 03:10:21.207006 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.207032 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 32 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.207090 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 32: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:21.207150 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.207206 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:21.207299 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250113 03:10:21.209278 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.209317 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 32 FOLLOWER]: Advancing to term 33
I20250113 03:10:21.209717 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.209831 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 33 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.209959 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 33 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.209959 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 33 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.210137 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.210213 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20250113 03:10:21.210847 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 32. Current term is 33. Ops: []
I20250113 03:10:21.211022 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.247543 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 32. Current term is 33. Ops: [32.16858-32.16858]
I20250113 03:10:21.247722 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 30.16857, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.252135 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 32. Current term is 33. Ops: [32.16858-32.16858]
I20250113 03:10:21.252297 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 30.16857, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.270589 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.270649 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 33 FOLLOWER]: Advancing to term 34
I20250113 03:10:21.271042 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.271143 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 34 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.271284 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 34 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.271281 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 34 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.271483 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.271561 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20250113 03:10:21.285283 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 34 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 32. Current term is 34. Ops: [32.16858-32.16858]
I20250113 03:10:21.285426 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 30.16857, Next index: 16858, Last known committed idx: 16857, Time since last communication: 0.000s
I20250113 03:10:21.288300 30096 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 LEADER]: Stepping down as leader of term 32
I20250113 03:10:21.288341 30096 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.288384 30096 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16858, Committed index: 16858, Last appended: 32.16858, Last appended by leader: 16858, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.288456 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 32 FOLLOWER]: Advancing to term 33
I20250113 03:10:21.304996 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.305038 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 34 FOLLOWER]: Advancing to term 35
I20250113 03:10:21.305375 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.305464 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 35 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.305595 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 35 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.305599 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 35 candidate_status { last_received { term: 30 index: 16857 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.305661 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 33 FOLLOWER]: Advancing to term 35
I20250113 03:10:21.306092 29835 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for term 35 because replica has last-logged OpId of term: 32 index: 16858, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16857.
I20250113 03:10:21.306236 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.306306 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20250113 03:10:21.314957 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 32 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:21.314998 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 32 FOLLOWER]: Advancing to term 33
I20250113 03:10:21.315341 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.315426 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 33 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.315552 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 16858 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.315558 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 33 candidate_status { last_received { term: 32 index: 16858 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.315618 29835 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 33. Current term is 35.
I20250113 03:10:21.315621 29683 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 33. Current term is 35.
I20250113 03:10:21.315783 29725 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 33 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 33. Current term is 35.
I20250113 03:10:21.315811 29725 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20250113 03:10:21.315833 29726 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 33 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 33. Current term is 35.
I20250113 03:10:21.315858 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 33 FOLLOWER]: Advancing to term 35
I20250113 03:10:21.316267 29935 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 35 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 33. Current term is 35.
I20250113 03:10:21.317903 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.317955 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 35 FOLLOWER]: Advancing to term 36
I20250113 03:10:21.318290 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.318362 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 36 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.318507 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 36 candidate_status { last_received { term: 32 index: 16858 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.318511 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 36 candidate_status { last_received { term: 32 index: 16858 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.318571 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 35 FOLLOWER]: Advancing to term 36
I20250113 03:10:21.318578 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 35 FOLLOWER]: Advancing to term 36
I20250113 03:10:21.319065 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 36.
I20250113 03:10:21.319065 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 36.
I20250113 03:10:21.319236 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:21.319300 30067 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 FOLLOWER]: Leader election won for term 36
I20250113 03:10:21.319344 30067 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.319376 30067 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16858, Committed index: 16858, Last appended: 32.16858, Last appended by leader: 16858, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.319671 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 32 to 36. New cstate: current_term: 36 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:21.342975 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 36 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 32 index: 16858. Preceding OpId from leader: term: 36 index: 16859. (index mismatch)
I20250113 03:10:21.343156 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16859, Last known committed idx: 16858, Time since last communication: 0.000s
I20250113 03:10:21.348970 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 36 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 30 index: 16857. Preceding OpId from leader: term: 36 index: 16859. (index mismatch)
I20250113 03:10:21.349150 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16859, Last known committed idx: 16857, Time since last communication: 0.000s
W20250113 03:10:21.351100 30096 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 33 current term: 36 is higher.
I20250113 03:10:21.379882 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:21.379925 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 36 FOLLOWER]: Advancing to term 37
I20250113 03:10:21.380264 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.380364 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 37 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.380551 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 36. Current term is 37. Ops: []
I20250113 03:10:21.380487 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 37 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.380513 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 37 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.380738 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.380746 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.380812 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20250113 03:10:21.393224 30096 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 34 current term: 36 is higher.
I20250113 03:10:21.410899 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 36. Current term is 37. Ops: []
I20250113 03:10:21.411073 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.416147 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.416193 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 37 FOLLOWER]: Advancing to term 38
I20250113 03:10:21.416553 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.416649 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 38 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.416760 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 38 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.416769 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 38 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.416929 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.417006 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20250113 03:10:21.442463 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 38 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 36. Current term is 38. Ops: []
I20250113 03:10:21.442615 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.449322 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.449368 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 38 FOLLOWER]: Advancing to term 39
I20250113 03:10:21.449753 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.449851 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 39 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.449954 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 39 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.449965 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 39 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.450145 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.450212 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20250113 03:10:21.459503 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 36 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:21.459550 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 36 FOLLOWER]: Advancing to term 37
I20250113 03:10:21.459918 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.460024 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 37 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.460126 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.460186 29683 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 37. Current term is 39.
I20250113 03:10:21.460139 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 37 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.460289 29726 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 37 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 37. Current term is 39.
I20250113 03:10:21.460315 29726 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20250113 03:10:21.460359 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 37 FOLLOWER]: Advancing to term 39
I20250113 03:10:21.460860 29935 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 39 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 37. Current term is 39.
I20250113 03:10:21.462635 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 39 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 36. Current term is 39. Ops: []
I20250113 03:10:21.462810 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.466759 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 36. Current term is 39. Ops: []
I20250113 03:10:21.466930 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.475795 30095 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 LEADER]: Stepping down as leader of term 36
I20250113 03:10:21.475836 30095 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.475876 30095 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16859, Committed index: 16859, Last appended: 36.16859, Last appended by leader: 16859, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.475946 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 36 FOLLOWER]: Advancing to term 37
I20250113 03:10:21.492192 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.492235 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 39 FOLLOWER]: Advancing to term 40
I20250113 03:10:21.492584 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.492686 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 40 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.492797 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 40 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.492810 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 40 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.492867 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 39 FOLLOWER]: Advancing to term 40
I20250113 03:10:21.492880 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 37 FOLLOWER]: Advancing to term 40
I20250113 03:10:21.493257 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 40.
I20250113 03:10:21.493276 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 40.
I20250113 03:10:21.493418 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:21.493491 29935 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 FOLLOWER]: Leader election won for term 40
I20250113 03:10:21.493539 29935 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.493582 29935 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16859, Committed index: 16859, Last appended: 36.16859, Last appended by leader: 16859, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.493888 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 36 to 40, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 40 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:21.519191 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 40 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 16859. Preceding OpId from leader: term: 40 index: 16860. (index mismatch)
I20250113 03:10:21.519362 29935 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.528884 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.528925 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 40 FOLLOWER]: Advancing to term 41
I20250113 03:10:21.529292 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.529377 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 41 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.529500 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 41 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.529520 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 41 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.529664 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.529743 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20250113 03:10:21.530114 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 40. Current term is 41. Ops: []
I20250113 03:10:21.530287 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.536192 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 40. Current term is 41. Ops: [40.16860-40.16860]
I20250113 03:10:21.536366 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.571022 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.571065 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 41 FOLLOWER]: Advancing to term 42
I20250113 03:10:21.571426 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.571508 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 42 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.571660 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 42 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.571730 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 40. Current term is 42. Ops: [40.16860-40.16860]
I20250113 03:10:21.571661 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 42 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.571907 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.571911 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 36.16859, Next index: 16860, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.571987 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250113 03:10:21.576187 29884 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 LEADER]: Stepping down as leader of term 40
I20250113 03:10:21.576221 29884 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.576267 29884 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16860, Committed index: 16860, Last appended: 40.16860, Last appended by leader: 16860, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.576349 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 40 FOLLOWER]: Advancing to term 41
I20250113 03:10:21.598798 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.598841 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 41 FOLLOWER]: Advancing to term 42
I20250113 03:10:21.599223 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.599315 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 42 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.599444 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.599463 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 42 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.599534 29835 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 42: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:21.599704 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:21.599781 29929 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250113 03:10:21.602414 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.602442 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 40 FOLLOWER]: Advancing to term 41
I20250113 03:10:21.602849 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.602941 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 41 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.603047 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 41 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.603047 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 41 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.603130 29760 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 41. Current term is 42.
I20250113 03:10:21.603138 29835 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 41. Current term is 42.
I20250113 03:10:21.603250 29650 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 41 election: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 41. Current term is 42.
I20250113 03:10:21.603277 29650 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20250113 03:10:21.603304 29649 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 41 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 41. Current term is 42.
I20250113 03:10:21.603329 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 41 FOLLOWER]: Advancing to term 42
I20250113 03:10:21.603682 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 41. Current term is 42.
I20250113 03:10:21.606336 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.606382 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 42 FOLLOWER]: Advancing to term 43
I20250113 03:10:21.606791 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.606874 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 43 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.607004 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 43 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.607004 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 43 candidate_status { last_received { term: 36 index: 16859 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.607086 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 42 FOLLOWER]: Advancing to term 43
I20250113 03:10:21.607092 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 42 FOLLOWER]: Advancing to term 43
I20250113 03:10:21.607578 29760 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for term 43 because replica has last-logged OpId of term: 40 index: 16860, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 16859.
I20250113 03:10:21.607578 29683 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for term 43 because replica has last-logged OpId of term: 40 index: 16860, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 16859.
I20250113 03:10:21.607764 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.607831 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20250113 03:10:21.630067 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.630118 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 43 FOLLOWER]: Advancing to term 44
I20250113 03:10:21.630511 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.630656 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 44 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.630769 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 44 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.630769 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 44 candidate_status { last_received { term: 40 index: 16860 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.630837 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 43 FOLLOWER]: Advancing to term 44
I20250113 03:10:21.630856 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 43 FOLLOWER]: Advancing to term 44
I20250113 03:10:21.631269 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 44.
I20250113 03:10:21.631273 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 44.
I20250113 03:10:21.631433 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:21.631511 29929 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 FOLLOWER]: Leader election won for term 44
I20250113 03:10:21.631567 29929 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.631613 29929 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16860, Committed index: 16860, Last appended: 40.16860, Last appended by leader: 16860, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.631919 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 40 to 44. New cstate: current_term: 44 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20250113 03:10:21.632941 29884 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 42 current term: 44 is higher.
I20250113 03:10:21.661233 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 44 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 36 index: 16859. Preceding OpId from leader: term: 44 index: 16861. (index mismatch)
I20250113 03:10:21.661417 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16861, Last known committed idx: 16859, Time since last communication: 0.000s
I20250113 03:10:21.662042 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 44 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 40 index: 16860. Preceding OpId from leader: term: 44 index: 16861. (index mismatch)
I20250113 03:10:21.662180 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16861, Last known committed idx: 16860, Time since last communication: 0.000s
I20250113 03:10:21.718866 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 44 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.718925 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 44 FOLLOWER]: Advancing to term 45
I20250113 03:10:21.719369 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.719465 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 45 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.719569 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 45 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.719569 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 45 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.719856 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.719926 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20250113 03:10:21.725397 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 45 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 44. Current term is 45. Ops: []
I20250113 03:10:21.725571 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 44.16861, Next index: 16862, Last known committed idx: 16861, Time since last communication: 0.000s
I20250113 03:10:21.750193 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.750231 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 45 FOLLOWER]: Advancing to term 46
I20250113 03:10:21.750602 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.750710 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 46 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.750855 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 46 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.750855 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 46 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.751078 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.751153 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20250113 03:10:21.760187 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 44. Current term is 46. Ops: []
I20250113 03:10:21.760363 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 44.16861, Next index: 16862, Last known committed idx: 16861, Time since last communication: 0.000s
I20250113 03:10:21.771440 29929 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 LEADER]: Stepping down as leader of term 44
I20250113 03:10:21.771492 29929 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.771564 29929 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16861, Committed index: 16861, Last appended: 44.16861, Last appended by leader: 16861, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.771634 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 44 FOLLOWER]: Advancing to term 45
I20250113 03:10:21.779500 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.779547 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 44 FOLLOWER]: Advancing to term 45
I20250113 03:10:21.779982 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.780089 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 45 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.780208 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 45 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.780208 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 45 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.780330 29835 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 45. Current term is 46.
I20250113 03:10:21.780478 29649 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 45 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 45. Current term is 46.
I20250113 03:10:21.780565 29649 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 45 election: Cancelling election due to peer responding with higher term
I20250113 03:10:21.780624 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 45 FOLLOWER]: Advancing to term 46
I20250113 03:10:21.780866 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 45.
I20250113 03:10:21.781165 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 46 FOLLOWER]: Leader election lost for term 45. Reason: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 45. Current term is 46.
I20250113 03:10:21.800583 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.800631 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 46 FOLLOWER]: Advancing to term 47
I20250113 03:10:21.801018 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.801117 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 47 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.801311 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 47 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.801364 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 46 FOLLOWER]: Advancing to term 47
I20250113 03:10:21.801312 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 47 candidate_status { last_received { term: 44 index: 16861 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.801440 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 45 FOLLOWER]: Advancing to term 47
I20250113 03:10:21.801828 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 47.
I20250113 03:10:21.801875 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 47.
I20250113 03:10:21.801988 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters:
I20250113 03:10:21.802055 30095 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 FOLLOWER]: Leader election won for term 47
I20250113 03:10:21.802105 30095 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.802150 30095 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16861, Committed index: 16861, Last appended: 44.16861, Last appended by leader: 16861, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.802500 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 44 to 47, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 47 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:21.829854 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 47 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 44 index: 16861. Preceding OpId from leader: term: 47 index: 16862. (index mismatch)
I20250113 03:10:21.830053 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16862, Last known committed idx: 16861, Time since last communication: 0.000s
I20250113 03:10:21.832903 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 47 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 44 index: 16861. Preceding OpId from leader: term: 47 index: 16862. (index mismatch)
I20250113 03:10:21.833071 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16862, Last known committed idx: 16861, Time since last communication: 0.000s
I20250113 03:10:21.861145 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 47 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:21.861197 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 47 FOLLOWER]: Advancing to term 48
I20250113 03:10:21.861586 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.861691 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 48 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.861805 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 48 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.861843 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 48 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.862016 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:21.862088 29929 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250113 03:10:21.863219 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 47. Current term is 48. Ops: []
I20250113 03:10:21.863390 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 47.16862, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.863665 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:21.863696 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 47 FOLLOWER]: Advancing to term 48
I20250113 03:10:21.864063 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.864171 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 48 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.864269 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 48 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.864305 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 48 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.864379 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 48: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:21.864506 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.864585 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250113 03:10:21.869297 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 48 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 47. Current term is 48. Ops: []
I20250113 03:10:21.869488 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 47.16862, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.891559 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 48 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 47. Current term is 48. Ops: []
I20250113 03:10:21.891764 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 47.16862, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.893589 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 47. Current term is 48. Ops: []
I20250113 03:10:21.893762 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 47.16862, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.894537 30095 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 LEADER]: Stepping down as leader of term 47
I20250113 03:10:21.894572 30095 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:21.894610 30095 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16862, Committed index: 16862, Last appended: 47.16862, Last appended by leader: 16862, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.894682 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 47 FOLLOWER]: Advancing to term 48
I20250113 03:10:21.894793 29929 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.894827 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 48 FOLLOWER]: Advancing to term 49
I20250113 03:10:21.895191 29929 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.895279 29929 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 49 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.895411 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 49 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.895469 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 48 FOLLOWER]: Advancing to term 49
I20250113 03:10:21.895411 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 49 candidate_status { last_received { term: 47 index: 16862 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.895500 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 48 FOLLOWER]: Advancing to term 49
I20250113 03:10:21.895881 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 49.
I20250113 03:10:21.895879 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 49.
I20250113 03:10:21.896052 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters:
I20250113 03:10:21.896126 29929 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 FOLLOWER]: Leader election won for term 49
I20250113 03:10:21.896176 29929 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:21.896219 29929 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16862, Committed index: 16862, Last appended: 47.16862, Last appended by leader: 16862, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.896505 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 47 to 49, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 49 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:21.922021 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 49 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 47 index: 16862. Preceding OpId from leader: term: 49 index: 16863. (index mismatch)
I20250113 03:10:21.922201 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.926966 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 49 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 47 index: 16862. Preceding OpId from leader: term: 49 index: 16863. (index mismatch)
I20250113 03:10:21.927167 29929 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16863, Last known committed idx: 16862, Time since last communication: 0.000s
I20250113 03:10:21.957790 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.957834 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 49 FOLLOWER]: Advancing to term 50
I20250113 03:10:21.958304 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.958418 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 50 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.958568 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 50 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.958580 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 50 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.958828 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.958915 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20250113 03:10:21.959934 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 49. Current term is 50. Ops: []
I20250113 03:10:21.960101 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 49.16863, Next index: 16864, Last known committed idx: 16863, Time since last communication: 0.000s
W20250113 03:10:21.960619 29907 lock_manager.cc:494] Txn 7 has not acquired the partition lock after 1000ms
W20250113 03:10:21.960623 29899 lock_manager.cc:494] Txn 7 has not acquired the partition lock after 1000ms
I20250113 03:10:21.961427 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 49. Current term is 50. Ops: [49.16864-49.16864]
I20250113 03:10:21.961606 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 49.16863, Next index: 16864, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:21.992147 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 49 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:21.992187 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 49 FOLLOWER]: Advancing to term 50
I20250113 03:10:21.992599 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.992693 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 50 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:21.992805 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 50 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.992836 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 50 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:21.992889 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 50: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:21.993047 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.993147 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20250113 03:10:21.993984 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 49. Current term is 50. Ops: [49.16864-49.16864]
I20250113 03:10:21.994171 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 49.16863, Next index: 16864, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:21.994387 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 50 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 49. Current term is 50. Ops: []
I20250113 03:10:21.994565 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 49.16864, Next index: 16865, Last known committed idx: 16864, Time since last communication: 0.000s
I20250113 03:10:21.995211 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:21.995254 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 50 FOLLOWER]: Advancing to term 51
I20250113 03:10:21.995836 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:21.996006 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 51 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:21.996129 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 51 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:21.996193 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 50 FOLLOWER]: Advancing to term 51
I20250113 03:10:21.996179 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 51 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:21.996636 29835 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for term 51 because replica has last-logged OpId of term: 49 index: 16864, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 16863.
I20250113 03:10:21.996780 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:21.996860 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20250113 03:10:22.008066 29929 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 LEADER]: Stepping down as leader of term 49
I20250113 03:10:22.008111 29929 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.008152 29929 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16864, Committed index: 16864, Last appended: 49.16864, Last appended by leader: 16864, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.008224 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 49 FOLLOWER]: Advancing to term 50
I20250113 03:10:22.031466 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.031513 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 50 FOLLOWER]: Advancing to term 51
I20250113 03:10:22.031939 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.032087 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 51 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.032193 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 51 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.032261 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 51: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:22.032225 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 51 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.032810 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 51.
I20250113 03:10:22.032958 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:22.033035 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 FOLLOWER]: Leader election won for term 51
I20250113 03:10:22.033088 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.033131 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16863, Committed index: 16863, Last appended: 49.16864, Last appended by leader: 16864, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.033474 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 49 to 51. New cstate: current_term: 51 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:22.044227 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.044273 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 51 FOLLOWER]: Advancing to term 52
I20250113 03:10:22.044672 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.044768 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 52 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.044859 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 52 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.044888 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 52 candidate_status { last_received { term: 49 index: 16863 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.044941 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 51 FOLLOWER]: Advancing to term 52
I20250113 03:10:22.045434 29835 raft_consensus.cc:2405] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for term 52 because replica has last-logged OpId of term: 49 index: 16864, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 16863.
I20250113 03:10:22.045559 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.045646 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20250113 03:10:22.062568 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 52 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 51. Current term is 52. Ops: []
I20250113 03:10:22.062758 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16865, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:22.064985 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 52 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 51. Current term is 52. Ops: []
I20250113 03:10:22.065129 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16865, Last known committed idx: 16864, Time since last communication: 0.000s
W20250113 03:10:22.075965 29929 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 50 current term: 51 is higher.
I20250113 03:10:22.096318 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 52 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 51. Current term is 52. Ops: [49.16864-51.16865]
I20250113 03:10:22.096488 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 49.16863, Next index: 16864, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:22.098860 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.098901 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 52 FOLLOWER]: Advancing to term 53
I20250113 03:10:22.099295 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.099375 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 53 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.099490 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 53 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.099517 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 53 candidate_status { last_received { term: 49 index: 16864 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.099561 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 52 FOLLOWER]: Advancing to term 53
I20250113 03:10:22.099984 29683 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 53.
I20250113 03:10:22.100147 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters: 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.100220 30095 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 FOLLOWER]: Leader election won for term 53
I20250113 03:10:22.100286 30095 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:22.100343 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 53 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 51. Current term is 53. Ops: [49.16864-51.16865]
I20250113 03:10:22.100332 30095 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16864, Committed index: 16864, Last appended: 49.16864, Last appended by leader: 16864, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.100473 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 LEADER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 51. Current term is 53. Ops: [51.16865-51.16865]
I20250113 03:10:22.100477 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 49.16863, Next index: 16864, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:22.100765 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 51 to 53, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 53 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:22.100993 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 49.16864, Next index: 16865, Last known committed idx: 16864, Time since last communication: 0.000s
I20250113 03:10:22.101063 29884 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer 773363024c05460c9bb9a4f875961edb log is divergent from this leader: its last log entry 53.16865 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16864
I20250113 03:10:22.101122 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 53 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 49 index: 16863. Preceding OpId from leader: term: 53 index: 16866. (index mismatch)
I20250113 03:10:22.101198 29760 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 LEADER]: Stepping down as leader of term 51
I20250113 03:10:22.101225 29760 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.101271 29760 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16863, Committed index: 16863, Last appended: 51.16865, Last appended by leader: 16865, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.101302 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16865, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:22.101336 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 51 FOLLOWER]: Advancing to term 53
I20250113 03:10:22.101970 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 53 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 51 index: 16865. Preceding OpId from leader: term: 53 index: 16866. (index mismatch)
I20250113 03:10:22.102098 30067 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16865, Last known committed idx: 16863, Time since last communication: 0.000s
I20250113 03:10:22.102174 30067 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer 8fa8ecf2b7614f1490b642f1b5eaee9a log is divergent from this leader: its last log entry 51.16865 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16863
I20250113 03:10:22.102368 29760 raft_consensus.cc:1212] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 53 FOLLOWER]: Deduplicated request from leader. Original: 49.16863->[49.16864-53.16866] Dedup: 49.16864->[53.16865-53.16866]
I20250113 03:10:22.102403 29760 pending_rounds.cc:77] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Aborting all ops after (but not including) 16864
I20250113 03:10:22.102418 29760 pending_rounds.cc:99] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Aborting uncommitted NO_OP operation due to leader change: 51.16865
I20250113 03:10:22.102442 29760 raft_consensus.cc:2883] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 03:10:22.133462 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 53 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.133504 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 53 FOLLOWER]: Advancing to term 54
I20250113 03:10:22.133924 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.134024 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 54 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.134141 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 54 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.134151 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 54 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.134311 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.134382 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20250113 03:10:22.139123 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 53. Current term is 54. Ops: []
I20250113 03:10:22.139295 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 53.16866, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
I20250113 03:10:22.166229 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 53. Current term is 54. Ops: []
I20250113 03:10:22.166379 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 53.16866, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
I20250113 03:10:22.172690 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.172744 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 54 FOLLOWER]: Advancing to term 55
I20250113 03:10:22.173133 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.173233 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 55 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.173343 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 55 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.173344 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 55 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.173529 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.173597 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20250113 03:10:22.199581 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 53. Current term is 55. Ops: []
I20250113 03:10:22.199759 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 53.16866, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
I20250113 03:10:22.228696 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 53. Current term is 55. Ops: []
I20250113 03:10:22.228838 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 53.16866, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
I20250113 03:10:22.235360 30096 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 LEADER]: Stepping down as leader of term 53
I20250113 03:10:22.235404 30096 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:22.235446 30096 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16866, Committed index: 16866, Last appended: 53.16866, Last appended by leader: 16866, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.235519 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 53 FOLLOWER]: Advancing to term 54
I20250113 03:10:22.238121 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.238168 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 55 FOLLOWER]: Advancing to term 56
I20250113 03:10:22.238528 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.238621 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 56 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.238739 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 56 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.238795 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 54 FOLLOWER]: Advancing to term 56
I20250113 03:10:22.238739 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 56 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.239208 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 56.
I20250113 03:10:22.239346 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:22.239416 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 FOLLOWER]: Leader election won for term 56
I20250113 03:10:22.239467 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.239506 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16866, Committed index: 16866, Last appended: 53.16866, Last appended by leader: 16866, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.239799 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 53 to 56, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 56 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20250113 03:10:22.253647 29929 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 53 current term: 56 is higher.
I20250113 03:10:22.258253 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 53 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.258296 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 53 FOLLOWER]: Advancing to term 54
I20250113 03:10:22.258689 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.258797 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 54 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.258906 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 54 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.258936 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 54 candidate_status { last_received { term: 53 index: 16866 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.258991 29835 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 54. Current term is 56.
I20250113 03:10:22.259091 29649 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 54 election: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 54. Current term is 56.
I20250113 03:10:22.259117 29649 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 54 election: Cancelling election due to peer responding with higher term
I20250113 03:10:22.259164 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 54 FOLLOWER]: Advancing to term 56
I20250113 03:10:22.259552 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 56 FOLLOWER]: Leader election lost for term 54. Reason: Vote denied by peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 54. Current term is 56.
I20250113 03:10:22.265002 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 53 index: 16866. Preceding OpId from leader: term: 56 index: 16867. (index mismatch)
I20250113 03:10:22.265192 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
I20250113 03:10:22.275071 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 56 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 53 index: 16866. Preceding OpId from leader: term: 56 index: 16867. (index mismatch)
I20250113 03:10:22.275260 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16867, Last known committed idx: 16866, Time since last communication: 0.000s
W20250113 03:10:22.281780 29929 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 53 current term: 56 is higher.
I20250113 03:10:22.306452 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:22.306497 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 56 FOLLOWER]: Advancing to term 57
I20250113 03:10:22.306833 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.306921 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 57 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.307053 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 57 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.307058 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 57 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.307245 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.307317 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20250113 03:10:22.310446 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 56. Current term is 57. Ops: []
I20250113 03:10:22.310621 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 56.16867, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.345520 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 56. Current term is 57. Ops: []
I20250113 03:10:22.345583 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.345647 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 57 FOLLOWER]: Advancing to term 58
I20250113 03:10:22.345716 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 56.16867, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.346048 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.346141 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 58 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.346246 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 58 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.346256 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 58 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.346443 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.346513 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20250113 03:10:22.353356 29884 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 LEADER]: Stepping down as leader of term 56
I20250113 03:10:22.353399 29884 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.353435 29884 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16867, Committed index: 16867, Last appended: 56.16867, Last appended by leader: 16867, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.353492 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 56 FOLLOWER]: Advancing to term 57
I20250113 03:10:22.372714 30067 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:22.372757 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 56 FOLLOWER]: Advancing to term 57
I20250113 03:10:22.373122 30067 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.373200 30067 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 57 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.373344 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 57 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.373348 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 57 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.373399 29683 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 57. Current term is 58.
I20250113 03:10:22.373528 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 57 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 57. Current term is 58.
I20250113 03:10:22.373556 29801 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 57 election: Cancelling election due to peer responding with higher term
I20250113 03:10:22.373605 30067 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 57 FOLLOWER]: Advancing to term 58
I20250113 03:10:22.373893 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 57.
I20250113 03:10:22.374030 30067 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 58 FOLLOWER]: Leader election lost for term 57. Reason: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 57. Current term is 58.
I20250113 03:10:22.384969 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.385015 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 58 FOLLOWER]: Advancing to term 59
I20250113 03:10:22.385423 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.385533 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 59 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.385654 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 59 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.385679 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 59 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.385725 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 57 FOLLOWER]: Advancing to term 59
I20250113 03:10:22.385730 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 58 FOLLOWER]: Advancing to term 59
I20250113 03:10:22.386122 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 59.
I20250113 03:10:22.386121 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 59.
I20250113 03:10:22.386265 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters:
I20250113 03:10:22.386343 30069 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 FOLLOWER]: Leader election won for term 59
I20250113 03:10:22.386384 30069 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.386411 30069 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16867, Committed index: 16867, Last appended: 56.16867, Last appended by leader: 16867, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.386731 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 56 to 59, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 59 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:22.420547 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.420619 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 59 FOLLOWER]: Advancing to term 60
I20250113 03:10:22.421020 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.421106 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 60 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.421253 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 60 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.421305 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 59 FOLLOWER]: Advancing to term 60
I20250113 03:10:22.421260 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 60 candidate_status { last_received { term: 56 index: 16867 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.421727 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 60.
I20250113 03:10:22.421860 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:22.421936 30096 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 FOLLOWER]: Leader election won for term 60
I20250113 03:10:22.421984 30096 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:22.422029 30096 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16867, Committed index: 16867, Last appended: 56.16867, Last appended by leader: 16867, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.422310 29581 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 59 to 60, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 60 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:22.422658 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 59. Current term is 60. Ops: []
I20250113 03:10:22.422776 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 59. Current term is 60. Ops: []
I20250113 03:10:22.422817 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.422967 30069 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer 773363024c05460c9bb9a4f875961edb log is divergent from this leader: its last log entry 60.16868 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16867
I20250113 03:10:22.423007 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.448259 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 59. Current term is 60. Ops: [59.16868-59.16868]
I20250113 03:10:22.448444 30116 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 56.16867, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.450688 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 56 index: 16867. Preceding OpId from leader: term: 60 index: 16868. (index mismatch)
I20250113 03:10:22.450870 30096 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.451112 29683 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 LEADER]: Stepping down as leader of term 59
I20250113 03:10:22.451153 29683 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.451206 29683 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16867, Committed index: 16867, Last appended: 59.16868, Last appended by leader: 16868, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.451289 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 59 FOLLOWER]: Advancing to term 60
I20250113 03:10:22.451709 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 60 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 59 index: 16868. Preceding OpId from leader: term: 60 index: 16868. (term mismatch)
I20250113 03:10:22.451751 29683 pending_rounds.cc:77] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting all ops after (but not including) 16867
I20250113 03:10:22.451761 29683 pending_rounds.cc:99] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Aborting uncommitted NO_OP operation due to leader change: 59.16868
I20250113 03:10:22.451773 29683 raft_consensus.cc:2883] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 60 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250113 03:10:22.451939 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16868, Last known committed idx: 16867, Time since last communication: 0.000s
I20250113 03:10:22.482398 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.482453 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 60 FOLLOWER]: Advancing to term 61
I20250113 03:10:22.482539 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 60 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.482587 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 60 FOLLOWER]: Advancing to term 61
I20250113 03:10:22.482904 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.482955 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.482996 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 61 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.483101 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 61 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.483129 29834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 61 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.483136 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 61 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.483107 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 61 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 60. Current term is 61. Ops: []
I20250113 03:10:22.483161 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 61 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.483244 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 61: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:22.483156 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 61 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.483354 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 60.16868, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.483364 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.483460 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.483461 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20250113 03:10:22.483551 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20250113 03:10:22.484795 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 60. Current term is 61. Ops: []
I20250113 03:10:22.484930 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 60.16868, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.513218 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 60. Current term is 61. Ops: []
I20250113 03:10:22.513422 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 60.16868, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.514751 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.514781 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 61 FOLLOWER]: Advancing to term 62
I20250113 03:10:22.515177 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.515280 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 62 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.515398 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 62 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.515426 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 62 candidate_status { last_received { term: 60 index: 16868 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.515486 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 61 FOLLOWER]: Advancing to term 62
I20250113 03:10:22.515516 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 60. Current term is 62. Ops: []
I20250113 03:10:22.515723 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 60.16868, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.515997 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 62.
I20250113 03:10:22.516139 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.516204 30069 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 FOLLOWER]: Leader election won for term 62
I20250113 03:10:22.516252 30069 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.516294 30069 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16868, Committed index: 16868, Last appended: 60.16868, Last appended by leader: 16868, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.516618 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 60 to 62, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 62 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
W20250113 03:10:22.539390 29940 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 60 current term: 62 is higher.
I20250113 03:10:22.545156 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 LEADER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 60. Current term is 62. Ops: []
I20250113 03:10:22.545336 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 60.16868, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.545396 30095 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer 52ecbe4b06b6444083ea4654c2c6cd33 log is divergent from this leader: its last log entry 62.16869 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16868
I20250113 03:10:22.548739 29835 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 LEADER]: Stepping down as leader of term 60
I20250113 03:10:22.548794 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 62 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 60 index: 16868. Preceding OpId from leader: term: 62 index: 16869. (index mismatch)
I20250113 03:10:22.548785 29835 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:22.548890 29835 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16868, Committed index: 16868, Last appended: 60.16868, Last appended by leader: 16868, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.548969 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 60 FOLLOWER]: Advancing to term 62
I20250113 03:10:22.548990 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.549546 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 62 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 60 index: 16868. Preceding OpId from leader: term: 62 index: 16869. (index mismatch)
I20250113 03:10:22.549696 29940 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16869, Last known committed idx: 16868, Time since last communication: 0.000s
I20250113 03:10:22.580364 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 62 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:22.580405 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 62 FOLLOWER]: Advancing to term 63
I20250113 03:10:22.580869 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.580971 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 63 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.581107 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 63 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.581107 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 63 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.581323 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.581413 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I20250113 03:10:22.584246 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 62. Current term is 63. Ops: []
I20250113 03:10:22.584416 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 62.16869, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.608563 30095 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 62 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:22.608608 30095 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 62 FOLLOWER]: Advancing to term 63
I20250113 03:10:22.609093 30095 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.609175 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 63 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 62. Current term is 63. Ops: []
I20250113 03:10:22.609190 30095 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 63 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.609342 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 63 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.609409 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb in current term 63: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:22.609380 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 63 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.609346 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 62.16869, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.609592 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.609656 30095 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I20250113 03:10:22.611781 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.611820 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 63 FOLLOWER]: Advancing to term 64
I20250113 03:10:22.612177 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.612260 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 64 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.612402 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 64 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.612406 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 64 candidate_status { last_received { term: 62 index: 16869 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.612476 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 63 FOLLOWER]: Advancing to term 64
I20250113 03:10:22.612869 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 64.
I20250113 03:10:22.613010 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:22.613085 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 FOLLOWER]: Leader election won for term 64
I20250113 03:10:22.613134 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.613176 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16869, Committed index: 16869, Last appended: 62.16869, Last appended by leader: 16869, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.613461 29580 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 62 to 64, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 64 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:22.619331 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 62. Current term is 64. Ops: []
I20250113 03:10:22.619489 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 62.16869, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.619556 30069 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer 8fa8ecf2b7614f1490b642f1b5eaee9a log is divergent from this leader: its last log entry 64.16870 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16869
I20250113 03:10:22.641479 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 62. Current term is 64. Ops: []
I20250113 03:10:22.641585 29683 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 LEADER]: Stepping down as leader of term 62
I20250113 03:10:22.641619 29683 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.641669 29683 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16869, Committed index: 16869, Last appended: 62.16869, Last appended by leader: 16869, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.641741 30116 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 62.16869, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.641813 30116 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Peer 8fa8ecf2b7614f1490b642f1b5eaee9a log is divergent from this leader: its last log entry 64.16870 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16869
I20250113 03:10:22.641868 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 62 FOLLOWER]: Advancing to term 64
I20250113 03:10:22.642311 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 64 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 62 index: 16869. Preceding OpId from leader: term: 64 index: 16870. (index mismatch)
I20250113 03:10:22.642505 29935 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.642622 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 64 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 62 index: 16869. Preceding OpId from leader: term: 64 index: 16870. (index mismatch)
I20250113 03:10:22.642781 29935 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16870, Last known committed idx: 16869, Time since last communication: 0.000s
I20250113 03:10:22.702690 30116 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 64 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:22.702745 30116 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 64 FOLLOWER]: Advancing to term 65
I20250113 03:10:22.703189 30116 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.703290 30116 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 65 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.703388 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 65 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.703408 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 65 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.703570 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.703648 30116 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20250113 03:10:22.706897 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 65 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 64. Current term is 65. Ops: []
I20250113 03:10:22.707079 29884 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 64.16870, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.734961 30116 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.735008 30116 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 65 FOLLOWER]: Advancing to term 66
I20250113 03:10:22.735394 30116 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.735489 30116 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 66 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.735644 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 66 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.735651 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 66 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.735842 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.735941 30116 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20250113 03:10:22.743304 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 64. Current term is 66. Ops: []
I20250113 03:10:22.743494 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 64.16870, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.752874 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 64 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:22.752921 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 64 FOLLOWER]: Advancing to term 65
I20250113 03:10:22.753325 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.753407 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 65 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.753582 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 65 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.753563 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 65 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.753646 29683 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 65. Current term is 66.
I20250113 03:10:22.753777 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 65 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 65. Current term is 66.
I20250113 03:10:22.753803 29801 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 65 election: Cancelling election due to peer responding with higher term
I20250113 03:10:22.753841 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 65 FOLLOWER]: Advancing to term 66
I20250113 03:10:22.754244 30096 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 66 FOLLOWER]: Leader election lost for term 65. Reason: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 65. Current term is 66.
I20250113 03:10:22.757280 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 66 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 64. Current term is 66. Ops: []
I20250113 03:10:22.757442 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 64.16870, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.773801 29683 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 64. Current term is 66. Ops: []
I20250113 03:10:22.773984 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 64.16870, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.782485 29884 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 LEADER]: Stepping down as leader of term 64
I20250113 03:10:22.782524 29884 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.782562 29884 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16870, Committed index: 16870, Last appended: 64.16870, Last appended by leader: 16870, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.782635 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 64 FOLLOWER]: Advancing to term 65
I20250113 03:10:22.784328 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.784370 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 66 FOLLOWER]: Advancing to term 67
I20250113 03:10:22.784924 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.785030 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 67 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.785153 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 67 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.785198 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 65 FOLLOWER]: Advancing to term 67
I20250113 03:10:22.785172 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 67 candidate_status { last_received { term: 64 index: 16870 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.785231 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 66 FOLLOWER]: Advancing to term 67
I20250113 03:10:22.785640 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 67.
I20250113 03:10:22.785655 29835 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in term 67.
I20250113 03:10:22.785782 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb; no voters:
I20250113 03:10:22.785869 30069 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 FOLLOWER]: Leader election won for term 67
I20250113 03:10:22.785921 30069 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 LEADER]: Becoming Leader. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.785969 30069 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16870, Committed index: 16870, Last appended: 64.16870, Last appended by leader: 16870, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.786324 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 reported cstate change: term changed from 64 to 67, leader changed from 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66) to 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65). New cstate: current_term: 67 leader_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:22.811578 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 67 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 64 index: 16870. Preceding OpId from leader: term: 67 index: 16871. (index mismatch)
I20250113 03:10:22.811774 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.816386 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 67 FOLLOWER]: Refusing update from remote peer 52ecbe4b06b6444083ea4654c2c6cd33: Log matching property violated. Preceding OpId in replica: term: 64 index: 16870. Preceding OpId from leader: term: 67 index: 16871. (index mismatch)
I20250113 03:10:22.816555 30069 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16871, Last known committed idx: 16870, Time since last communication: 0.000s
I20250113 03:10:22.842945 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 67 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:22.842988 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 67 FOLLOWER]: Advancing to term 68
I20250113 03:10:22.843390 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.843482 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 68 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.843618 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 68 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.843637 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 68 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.843832 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.843902 30096 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20250113 03:10:22.847173 29935 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 67 FOLLOWER]: Starting leader election (detected failure of leader 52ecbe4b06b6444083ea4654c2c6cd33)
I20250113 03:10:22.847208 29935 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 67 FOLLOWER]: Advancing to term 68
I20250113 03:10:22.847602 29935 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.847719 29935 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 68 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.847863 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 68 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.847864 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 68 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.847962 29835 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 68: Already voted for candidate 773363024c05460c9bb9a4f875961edb in this term.
I20250113 03:10:22.848080 29725 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.848169 29935 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20250113 03:10:22.849036 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 68 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 67. Current term is 68. Ops: []
I20250113 03:10:22.849177 30069 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 67.16871, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.849287 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 67. Current term is 68. Ops: []
I20250113 03:10:22.849474 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 67.16871, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.877498 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 68 FOLLOWER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 67. Current term is 68. Ops: []
I20250113 03:10:22.877677 30116 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 67.16871, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.882189 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.882226 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 68 FOLLOWER]: Advancing to term 69
I20250113 03:10:22.882591 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.882681 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 69 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:22.882819 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 69 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.882827 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 69 candidate_status { last_received { term: 67 index: 16871 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.882874 29760 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 68 FOLLOWER]: Advancing to term 69
I20250113 03:10:22.883348 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 69.
I20250113 03:10:22.883488 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33
I20250113 03:10:22.883553 30096 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 FOLLOWER]: Leader election won for term 69
I20250113 03:10:22.883597 30096 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 LEADER]: Becoming Leader. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:22.883627 30096 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16871, Committed index: 16871, Last appended: 67.16871, Last appended by leader: 16871, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.883958 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb reported cstate change: term changed from 67 to 69, leader changed from 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65) to 773363024c05460c9bb9a4f875961edb (127.20.18.67). New cstate: current_term: 69 leader_uuid: "773363024c05460c9bb9a4f875961edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:22.885527 29835 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 LEADER]: Rejecting Update request from peer 52ecbe4b06b6444083ea4654c2c6cd33 for earlier term 67. Current term is 69. Ops: []
I20250113 03:10:22.885695 29940 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: INVALID_TERM, Last received: 67.16871, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.885757 29940 consensus_queue.cc:1225] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [LEADER]: Peer 773363024c05460c9bb9a4f875961edb log is divergent from this leader: its last log entry 69.16872 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16871
I20250113 03:10:22.895866 30069 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 LEADER]: Stepping down as leader of term 67
I20250113 03:10:22.895900 30069 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Running, Role: LEADER
I20250113 03:10:22.895936 30069 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16871, Committed index: 16871, Last appended: 67.16871, Last appended by leader: 16871, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.895998 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 67 FOLLOWER]: Advancing to term 68
I20250113 03:10:22.912084 29683 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 68 FOLLOWER]: Advancing to term 69
I20250113 03:10:22.912093 29760 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 69 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 67 index: 16871. Preceding OpId from leader: term: 69 index: 16872. (index mismatch)
I20250113 03:10:22.912287 30096 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.912632 29683 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 69 FOLLOWER]: Refusing update from remote peer 773363024c05460c9bb9a4f875961edb: Log matching property violated. Preceding OpId in replica: term: 67 index: 16871. Preceding OpId from leader: term: 69 index: 16872. (index mismatch)
I20250113 03:10:22.912834 30095 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16872, Last known committed idx: 16871, Time since last communication: 0.000s
I20250113 03:10:22.943259 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 69 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.943300 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 69 FOLLOWER]: Advancing to term 70
I20250113 03:10:22.943699 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.943780 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 69. Current term is 70. Ops: []
I20250113 03:10:22.943805 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 70 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.943902 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 70 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.943917 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 70 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.943918 29940 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 69 FOLLOWER]: Starting leader election (detected failure of leader 773363024c05460c9bb9a4f875961edb)
I20250113 03:10:22.944002 29940 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 69 FOLLOWER]: Advancing to term 70
I20250113 03:10:22.944061 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 69.16872, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:22.944516 29940 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.944617 29683 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in current term 70: Already voted for candidate 52ecbe4b06b6444083ea4654c2c6cd33 in this term.
I20250113 03:10:22.944633 29940 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 70 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.944700 29684 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 69. Current term is 70. Ops: []
I20250113 03:10:22.944729 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 70 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:22.944744 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 70 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.944749 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 70 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.944777 29760 raft_consensus.cc:2388] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ecbe4b06b6444083ea4654c2c6cd33 in current term 70: Already voted for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in this term.
I20250113 03:10:22.944886 30095 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 69.16872, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:22.944993 29884 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Leader election lost for term 70. Reason: could not achieve majority
I20250113 03:10:22.945092 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 70 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:22.945158 29940 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Leader election lost for term 70. Reason: could not achieve majority
I20250113 03:10:22.975728 29760 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 69. Current term is 70. Ops: []
I20250113 03:10:22.975909 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 }, Status: INVALID_TERM, Last received: 69.16872, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:22.977948 29684 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 69. Current term is 70. Ops: []
I20250113 03:10:22.978132 30096 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 69.16872, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:22.980258 29884 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:22.980302 29884 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 70 FOLLOWER]: Advancing to term 71
I20250113 03:10:22.980767 29884 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.980883 29884 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 71 election: Requested vote from peers 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:22.980998 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 71 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:22.981005 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" candidate_term: 71 candidate_status { last_received { term: 69 index: 16872 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:22.981078 29684 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 70 FOLLOWER]: Advancing to term 71
I20250113 03:10:22.981582 29684 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa8ecf2b7614f1490b642f1b5eaee9a in term 71.
I20250113 03:10:22.981732 29726 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a; no voters: 773363024c05460c9bb9a4f875961edb
I20250113 03:10:22.981815 29884 raft_consensus.cc:2798] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 FOLLOWER]: Leader election won for term 71
I20250113 03:10:22.981858 29884 raft_consensus.cc:695] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 LEADER]: Becoming Leader. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:22.981885 29884 consensus_queue.cc:237] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16872, Committed index: 16872, Last appended: 69.16872, Last appended by leader: 16872, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:22.982198 29578 catalog_manager.cc:5526] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a reported cstate change: term changed from 69 to 71, leader changed from 773363024c05460c9bb9a4f875961edb (127.20.18.67) to 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66). New cstate: current_term: 71 leader_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:23.005211 29684 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 71 FOLLOWER]: Rejecting Update request from peer 773363024c05460c9bb9a4f875961edb for earlier term 69. Current term is 71. Ops: []
I20250113 03:10:23.005373 30067 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 69.16872, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:23.006389 29835 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 LEADER]: Stepping down as leader of term 69
I20250113 03:10:23.006420 29835 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Running, Role: LEADER
I20250113 03:10:23.006457 29835 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16872, Committed index: 16872, Last appended: 69.16872, Last appended by leader: 16872, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.006512 29835 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 69 FOLLOWER]: Advancing to term 71
I20250113 03:10:23.006916 29835 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 71 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 69 index: 16872. Preceding OpId from leader: term: 71 index: 16873. (index mismatch)
I20250113 03:10:23.007091 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:23.019351 29684 raft_consensus.cc:1270] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 71 FOLLOWER]: Refusing update from remote peer 8fa8ecf2b7614f1490b642f1b5eaee9a: Log matching property violated. Preceding OpId in replica: term: 69 index: 16872. Preceding OpId from leader: term: 71 index: 16873. (index mismatch)
I20250113 03:10:23.019526 29884 consensus_queue.cc:1035] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16873, Last known committed idx: 16872, Time since last communication: 0.000s
I20250113 03:10:23.050302 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 71 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:23.050345 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 71 FOLLOWER]: Advancing to term 72
I20250113 03:10:23.050742 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.050837 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 72 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:23.050983 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 72 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:23.050995 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 72 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:23.051174 29650 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:23.051242 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I20250113 03:10:23.054843 29684 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 71. Current term is 72. Ops: []
I20250113 03:10:23.055008 29929 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 71.16873, Next index: 16874, Last known committed idx: 16873, Time since last communication: 0.000s
I20250113 03:10:23.078140 29684 raft_consensus.cc:1235] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Rejecting Update request from peer 8fa8ecf2b7614f1490b642f1b5eaee9a for earlier term 71. Current term is 72. Ops: []
I20250113 03:10:23.078305 29935 consensus_queue.cc:1046] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 }, Status: INVALID_TERM, Last received: 71.16873, Next index: 16874, Last known committed idx: 16873, Time since last communication: 0.000s
I20250113 03:10:23.087713 30069 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:23.087756 30069 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 72 FOLLOWER]: Advancing to term 73
I20250113 03:10:23.088121 30069 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.088209 30069 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 73 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
I20250113 03:10:23.088337 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 73 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:23.088338 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" candidate_term: 73 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "773363024c05460c9bb9a4f875961edb"
I20250113 03:10:23.088537 29649 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 73 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:23.088605 30069 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Leader election lost for term 73. Reason: could not achieve majority
I20250113 03:10:23.092149 29929 raft_consensus.cc:3049] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 LEADER]: Stepping down as leader of term 71
I20250113 03:10:23.092178 29929 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Running, Role: LEADER
I20250113 03:10:23.092212 29929 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16873, Committed index: 16873, Last appended: 71.16873, Last appended by leader: 16873, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.092288 29929 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 71 FOLLOWER]: Advancing to term 72
I20250113 03:10:23.092343 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 71 FOLLOWER]: Starting leader election (detected failure of leader 8fa8ecf2b7614f1490b642f1b5eaee9a)
I20250113 03:10:23.092378 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 71 FOLLOWER]: Advancing to term 72
I20250113 03:10:23.092834 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.092921 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 72 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
I20250113 03:10:23.093060 29684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 72 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
I20250113 03:10:23.093060 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959b6ebdb4cc463f8a3bcae242eb4bd9" candidate_uuid: "773363024c05460c9bb9a4f875961edb" candidate_term: 72 candidate_status { last_received { term: 71 index: 16873 } } ignore_live_leader: false dest_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
I20250113 03:10:23.093142 29684 raft_consensus.cc:2363] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 72. Current term is 73.
I20250113 03:10:23.093276 29801 leader_election.cc:400] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 72 election: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 72. Current term is 73.
I20250113 03:10:23.093312 29801 leader_election.cc:403] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 72 election: Cancelling election due to peer responding with higher term
I20250113 03:10:23.093365 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 72 FOLLOWER]: Advancing to term 73
I20250113 03:10:23.093607 29760 raft_consensus.cc:2463] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773363024c05460c9bb9a4f875961edb in term 72.
I20250113 03:10:23.093854 30096 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 73 FOLLOWER]: Leader election lost for term 72. Reason: Vote denied by peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087) with higher term. Message: Invalid argument: T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 773363024c05460c9bb9a4f875961edb for earlier term 72. Current term is 73.
W20250113 03:10:23.101022 29899 lock_manager.cc:494] Txn 8 has not acquired the partition lock after 1000ms
I20250113 03:10:23.102432 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:10:23.104470 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:23.104565 20553 tablet_replica.cc:331] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: stopping tablet replica
I20250113 03:10:23.104611 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:23.104646 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Raft consensus is shut down!
W20250113 03:10:23.107486 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:23.119294 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:10:23.119937 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:10:23.121919 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:23.122001 20553 tablet_replica.cc:331] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: stopping tablet replica
I20250113 03:10:23.122045 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Raft consensus shutting down.
W20250113 03:10:23.134155 29884 raft_consensus.cc:668] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping [suppressed 3 similar messages]
I20250113 03:10:23.136962 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:23.137542 30096 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:23.137583 30096 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 73 FOLLOWER]: Advancing to term 74
I20250113 03:10:23.137944 30096 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:23.138027 30096 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 74 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:23.138231 29801 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250113 03:10:23.138504 29801 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 74 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250113 03:10:23.138549 29801 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 74 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:23.138574 29801 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:23.138636 30096 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
W20250113 03:10:23.147538 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:23.149106 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:10:23.149674 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:10:23.152774 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:23.152861 20553 tablet_replica.cc:331] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: stopping tablet replica
I20250113 03:10:23.152909 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:23.161015 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Raft consensus is shut down!
W20250113 03:10:23.163173 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:23.164603 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:10:23.165259 20553 master.cc:537] Master@127.20.18.126:46381 shutting down...
I20250113 03:10:23.167042 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:10:23.167090 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:23.167124 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: stopping tablet replica
I20250113 03:10:23.178447 20553 master.cc:559] Master@127.20.18.126:46381 shutdown complete.
I20250113 03:10:23.179355 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:23.180717 30148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.181388 30149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:23.182205 20553 server_base.cc:1034] running on GCE node
W20250113 03:10:23.182865 30151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:23.182986 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:23.183022 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:23.183041 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737823183041 us; error 0 us; skew 500 ppm
I20250113 03:10:23.187179 20553 webserver.cc:458] Webserver started at http://127.20.18.126:34189/ using document root <none> and password file <none>
I20250113 03:10:23.187254 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:23.187290 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:10:23.187727 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.188110 30156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.188220 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.188257 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "7bec902c16a04168b317d0c3f9266f0f"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:23.188303 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:23.199173 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:23.199299 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:23.201335 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:46381
I20250113 03:10:23.202701 30209 sys_catalog.cc:263] Verifying existing consensus state
I20250113 03:10:23.208410 30208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:46381 every 8 connection(s)
I20250113 03:10:23.209693 30209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Bootstrap starting.
I20250113 03:10:23.213770 30209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=37 ignored=0} inserts{seen=5 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:10:23.213973 30209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Bootstrap complete.
I20250113 03:10:23.214105 30209 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:23.214144 30209 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bec902c16a04168b317d0c3f9266f0f, State: Initialized, Role: FOLLOWER
I20250113 03:10:23.214177 30209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:23.214211 30209 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:10:23.214228 30209 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:10:23.214251 30209 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:23.214630 30209 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:23.214685 30209 leader_election.cc:304] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bec902c16a04168b317d0c3f9266f0f; no voters:
I20250113 03:10:23.214802 30209 leader_election.cc:290] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250113 03:10:23.214845 30212 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 2 FOLLOWER]: Leader election won for term 2
I20250113 03:10:23.214931 30209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:10:23.214975 30212 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 2 LEADER]: Becoming Leader. State: Replica: 7bec902c16a04168b317d0c3f9266f0f, State: Running, Role: LEADER
I20250113 03:10:23.215022 30212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER }
I20250113 03:10:23.215181 30212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bec902c16a04168b317d0c3f9266f0f. Latest consensus state: current_term: 2 leader_uuid: "7bec902c16a04168b317d0c3f9266f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER } }
I20250113 03:10:23.215235 30212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:23.215201 30214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7bec902c16a04168b317d0c3f9266f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bec902c16a04168b317d0c3f9266f0f" member_type: VOTER } }
I20250113 03:10:23.215342 30214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:23.215454 30217 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:10:23.215736 30217 catalog_manager.cc:670] Loaded metadata for table test-workload [id=672a7a4ce6ac490eb281119f1e6fe6c7]
I20250113 03:10:23.215902 30217 tablet_loader.cc:96] loaded metadata for tablet 959b6ebdb4cc463f8a3bcae242eb4bd9 (table test-workload [id=672a7a4ce6ac490eb281119f1e6fe6c7])
I20250113 03:10:23.215989 30217 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:10:23.216118 30217 catalog_manager.cc:1260] Loaded cluster ID: 10ac3db4a2cf46399e077d114d6bcc23
I20250113 03:10:23.216176 30217 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:10:23.216219 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:10:23.216898 30217 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:10:23.217006 30217 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f: Loaded TSK: 0
I20250113 03:10:23.217079 30217 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:10:23.217417 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:23.218348 30228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.218936 30229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.219487 30231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:23.220055 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:23.220135 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:23.220156 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:23.220166 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737823220167 us; error 0 us; skew 500 ppm
I20250113 03:10:23.220607 20553 webserver.cc:458] Webserver started at http://127.20.18.65:34941/ using document root <none> and password file <none>
I20250113 03:10:23.220682 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:23.220710 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:10:23.221108 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.221480 30236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.221580 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.221616 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "52ecbe4b06b6444083ea4654c2c6cd33"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:23.221657 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:23.238852 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:23.239014 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:23.239104 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:23.239274 30243 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250113 03:10:23.239744 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:10:23.239775 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250113 03:10:23.239794 20553 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 03:10:23.240099 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:10:23.240121 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.240135 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap starting.
I20250113 03:10:23.243117 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:37977
I20250113 03:10:23.243187 30302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:37977 every 8 connection(s)
I20250113 03:10:23.243376 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:23.244640 30307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.244665 30308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.245193 30310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:23.245908 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:23.246028 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:23.246060 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:23.246075 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737823246075 us; error 0 us; skew 500 ppm
I20250113 03:10:23.246338 30303 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:23.246384 30303 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:23.246465 30303 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
I20250113 03:10:23.246634 20553 webserver.cc:458] Webserver started at http://127.20.18.66:35655/ using document root <none> and password file <none>
I20250113 03:10:23.246676 30173 ts_manager.cc:194] Registered new tserver with Master: 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:37977)
I20250113 03:10:23.246711 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:23.246734 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:10:23.247233 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.247368 30173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46136
I20250113 03:10:23.247740 30315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.247843 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.247889 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:23.247925 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:23.276655 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:23.276801 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:23.276916 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:23.277496 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:10:23.277537 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.277837 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:10:23.277874 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.277923 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap starting.
I20250113 03:10:23.281297 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:36459
I20250113 03:10:23.281687 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:10:23.282671 30380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:36459 every 8 connection(s)
W20250113 03:10:23.285709 30383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.286195 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:23.286888 30384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:23.287482 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:23.287487 30381 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:23.287583 30381 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:23.287626 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:23.287647 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:23.287662 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737823287661 us; error 0 us; skew 500 ppm
I20250113 03:10:23.287680 30381 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
I20250113 03:10:23.288218 30173 ts_manager.cc:194] Registered new tserver with Master: 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:36459)
I20250113 03:10:23.288513 30173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46148
I20250113 03:10:23.288232 20553 webserver.cc:458] Webserver started at http://127.20.18.67:46201/ using document root <none> and password file <none>
I20250113 03:10:23.288687 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:23.288712 20553 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250113 03:10:23.292697 20553 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.000s
I20250113 03:10:23.293119 30392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.293218 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.293248 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "773363024c05460c9bb9a4f875961edb"
format_stamp: "Formatted at 2025-01-13 03:10:09 on dist-test-slave-d4dl"
I20250113 03:10:23.293284 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:23.322937 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:23.323127 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:23.323231 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:23.323863 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 03:10:23.323904 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.324203 20553 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 03:10:23.324239 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:23.324301 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap starting.
I20250113 03:10:23.327898 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34769
I20250113 03:10:23.336707 30457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34769 every 8 connection(s)
I20250113 03:10:23.340080 30458 heartbeater.cc:346] Connected to a master server at 127.20.18.126:46381
I20250113 03:10:23.340133 30458 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:23.340225 30458 heartbeater.cc:510] Master 127.20.18.126:46381 requested a full tablet report, sending...
I20250113 03:10:23.340437 30173 ts_manager.cc:194] Registered new tserver with Master: 773363024c05460c9bb9a4f875961edb (127.20.18.67:34769)
I20250113 03:10:23.340909 30173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46164
I20250113 03:10:23.341004 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012953738s
I20250113 03:10:23.922859 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap replayed 1/4 log segments. Stats: ops{read=4976 overwritten=0 applied=4973 ignored=0} inserts{seen=248600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:10:24.247859 30303 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:24.273630 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap replayed 1/4 log segments. Stats: ops{read=4968 overwritten=1 applied=4965 ignored=0} inserts{seen=248200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:24.277343 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap replayed 1/4 log segments. Stats: ops{read=4632 overwritten=0 applied=4630 ignored=0} inserts{seen=231450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:24.289004 30381 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:24.341827 30458 heartbeater.cc:502] Master 127.20.18.126:46381 was elected leader, sending a full tablet report...
I20250113 03:10:24.557055 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap replayed 2/4 log segments. Stats: ops{read=9937 overwritten=0 applied=9935 ignored=0} inserts{seen=496700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:25.143790 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap replayed 3/4 log segments. Stats: ops{read=14887 overwritten=0 applied=14884 ignored=0} inserts{seen=744150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:10:25.282404 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap replayed 2/4 log segments. Stats: ops{read=9262 overwritten=0 applied=9260 ignored=0} inserts{seen=462950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:25.349548 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap replayed 2/4 log segments. Stats: ops{read=9938 overwritten=1 applied=9935 ignored=0} inserts{seen=496700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 03:10:25.376500 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap replayed 4/4 log segments. Stats: ops{read=16873 overwritten=0 applied=16873 ignored=1} inserts{seen=841902 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:10:25.376943 30399 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Bootstrap complete.
I20250113 03:10:25.382021 30399 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Time spent bootstrapping tablet: real 2.058s user 1.673s sys 0.379s
I20250113 03:10:25.382148 30399 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.382212 30399 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773363024c05460c9bb9a4f875961edb, State: Initialized, Role: FOLLOWER
I20250113 03:10:25.382258 30399 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16873, Last appended: 71.16873, Last appended by leader: 16873, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.382418 30399 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:25.408586 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.408643 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 74 FOLLOWER]: Advancing to term 75
I20250113 03:10:25.409368 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.409516 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 75 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.410231 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 75 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.410284 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 75 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.410305 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.410391 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20250113 03:10:25.445729 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.445780 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 75 FOLLOWER]: Advancing to term 76
I20250113 03:10:25.446358 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.446465 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 76 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.446957 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 76 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.447006 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 76 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.447031 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.447093 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20250113 03:10:25.490415 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.490468 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 76 FOLLOWER]: Advancing to term 77
I20250113 03:10:25.491027 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.491139 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 77 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.491597 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 77 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.491659 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 77 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.491680 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.491766 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20250113 03:10:25.542266 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.542325 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 77 FOLLOWER]: Advancing to term 78
I20250113 03:10:25.542912 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.543027 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 78 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.543463 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 78 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.543517 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 78 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.543538 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.543597 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I20250113 03:10:25.656417 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 78 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.656499 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 78 FOLLOWER]: Advancing to term 79
I20250113 03:10:25.657183 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.657315 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 79 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.657848 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 79 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.657905 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 79 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.657923 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.658018 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20250113 03:10:25.712551 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.712610 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 79 FOLLOWER]: Advancing to term 80
I20250113 03:10:25.713205 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.713322 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 80 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.713822 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 80 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.713888 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 80 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.713915 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 80 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.713963 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 80 FOLLOWER]: Leader election lost for term 80. Reason: could not achieve majority
I20250113 03:10:25.789494 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 80 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:25.789553 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 80 FOLLOWER]: Advancing to term 81
I20250113 03:10:25.790160 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:25.790272 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 81 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:25.790817 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 81 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:25.790879 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 81 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:25.790901 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:25.790992 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20250113 03:10:25.893647 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap replayed 3/4 log segments. Stats: ops{read=13893 overwritten=0 applied=13890 ignored=0} inserts{seen=694450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250113 03:10:25.963339 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap replayed 3/4 log segments. Stats: ops{read=14899 overwritten=1 applied=14897 ignored=0} inserts{seen=744800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 03:10:26.040619 30469 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:26.040683 30469 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 81 FOLLOWER]: Advancing to term 82
I20250113 03:10:26.041363 30469 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.041517 30469 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 82 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087)
W20250113 03:10:26.042078 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 82 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
W20250113 03:10:26.042131 30396 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 82 election: RPC error from VoteRequest() call to peer 52ecbe4b06b6444083ea4654c2c6cd33 (127.20.18.65:43087): Network error: Client connection negotiation failed: client connection to 127.20.18.65:43087: connect: Connection refused (error 111)
I20250113 03:10:26.042155 30396 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [CANDIDATE]: Term 82 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 773363024c05460c9bb9a4f875961edb; no voters: 52ecbe4b06b6444083ea4654c2c6cd33, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:26.042215 30469 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 82 FOLLOWER]: Leader election lost for term 82. Reason: could not achieve majority
I20250113 03:10:26.196740 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap replayed 4/4 log segments. Stats: ops{read=16876 overwritten=3 applied=16873 ignored=1} inserts{seen=841902 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:10:26.197109 30243 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Bootstrap complete.
I20250113 03:10:26.202214 30243 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Time spent bootstrapping tablet: real 2.962s user 2.583s sys 0.375s
I20250113 03:10:26.202351 30243 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.202409 30243 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ecbe4b06b6444083ea4654c2c6cd33, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.202454 30243 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16873, Last appended: 71.16873, Last appended by leader: 16873, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.202620 30243 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250113 03:10:26.232787 30475 raft_consensus.cc:491] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:26.232834 30475 raft_consensus.cc:3054] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 73 FOLLOWER]: Advancing to term 74
I20250113 03:10:26.233361 30475 raft_consensus.cc:513] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.233497 30475 leader_election.cc:290] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 74 election: Requested vote from peers 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093), 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757)
W20250113 03:10:26.234126 30239 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 74 election: RPC error from VoteRequest() call to peer 773363024c05460c9bb9a4f875961edb (127.20.18.67:46757): Network error: Client connection negotiation failed: client connection to 127.20.18.67:46757: connect: Connection refused (error 111)
W20250113 03:10:26.234201 30240 leader_election.cc:336] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 74 election: RPC error from VoteRequest() call to peer 8fa8ecf2b7614f1490b642f1b5eaee9a (127.20.18.66:35093): Network error: Client connection negotiation failed: client connection to 127.20.18.66:35093: connect: Connection refused (error 111)
I20250113 03:10:26.234256 30240 leader_election.cc:304] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ecbe4b06b6444083ea4654c2c6cd33; no voters: 773363024c05460c9bb9a4f875961edb, 8fa8ecf2b7614f1490b642f1b5eaee9a
I20250113 03:10:26.234314 30475 raft_consensus.cc:2743] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
I20250113 03:10:26.247905 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap replayed 4/4 log segments. Stats: ops{read=16874 overwritten=1 applied=16873 ignored=1} inserts{seen=841902 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 03:10:26.248198 30322 tablet_bootstrap.cc:492] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Bootstrap complete.
I20250113 03:10:26.253290 30322 ts_tablet_manager.cc:1397] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Time spent bootstrapping tablet: real 2.975s user 2.537s sys 0.436s
I20250113 03:10:26.253407 30322 raft_consensus.cc:357] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.253451 30322 raft_consensus.cc:738] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa8ecf2b7614f1490b642f1b5eaee9a, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.253480 30322 consensus_queue.cc:260] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16873, Last appended: 71.16873, Last appended by leader: 16873, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fa8ecf2b7614f1490b642f1b5eaee9a" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 35093 } } peers { permanent_uuid: "52ecbe4b06b6444083ea4654c2c6cd33" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43087 } } peers { permanent_uuid: "773363024c05460c9bb9a4f875961edb" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 46757 } }
I20250113 03:10:26.253602 30322 ts_tablet_manager.cc:1428] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:26.253733 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:10:26.255499 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:26.255625 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:10:26.255690 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 74 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:26.255728 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 52ecbe4b06b6444083ea4654c2c6cd33 [term 74 FOLLOWER]: Raft consensus is shut down!
W20250113 03:10:26.257791 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:26.268942 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:10:26.269650 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:10:26.271915 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:26.272043 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:10:26.272094 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:26.272135 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 8fa8ecf2b7614f1490b642f1b5eaee9a [term 72 FOLLOWER]: Raft consensus is shut down!
W20250113 03:10:26.281069 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:26.292184 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:10:26.292791 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:10:26.294806 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:26.294903 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:10:26.294960 20553 raft_consensus.cc:2238] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 82 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:26.294987 20553 raft_consensus.cc:2267] T 959b6ebdb4cc463f8a3bcae242eb4bd9 P 773363024c05460c9bb9a4f875961edb [term 82 FOLLOWER]: Raft consensus is shut down!
W20250113 03:10:26.304620 20553 mvcc.cc:118] aborting op with timestamp 7113678119324905472 in state 0; MVCC is closed
I20250113 03:10:26.315752 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:10:26.316416 20553 master.cc:537] Master@127.20.18.126:46381 shutting down...
I20250113 03:10:26.318094 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 2 LEADER]: Raft consensus shutting down.
I20250113 03:10:26.318138 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7bec902c16a04168b317d0c3f9266f0f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:26.318163 20553 tablet_replica.cc:331] stopping tablet replica
I20250113 03:10:26.329197 20553 master.cc:559] Master@127.20.18.126:46381 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestFrequentElections (16873 ms)
[ RUN ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm
I20250113 03:10:26.348512 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:33433
I20250113 03:10:26.348650 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:26.349767 30480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.350585 30481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.351290 30483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:26.351969 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:26.352051 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:26.352082 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:26.352094 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737826352095 us; error 0 us; skew 500 ppm
I20250113 03:10:26.352682 20553 webserver.cc:458] Webserver started at http://127.20.18.126:36793/ using document root <none> and password file <none>
I20250113 03:10:26.352741 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:26.352782 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:26.352826 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:26.353042 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "37ed2653247a48ae87402267c04c3c96"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.353729 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:26.354066 30488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.354169 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.354208 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "37ed2653247a48ae87402267c04c3c96"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.354243 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:26.365507 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:26.365625 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:26.367539 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:33433
I20250113 03:10:26.371033 30540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:33433 every 8 connection(s)
I20250113 03:10:26.371135 30541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:26.371886 30541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96: Bootstrap starting.
I20250113 03:10:26.372112 30541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:26.372449 30541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96: No bootstrap required, opened a new log
I20250113 03:10:26.372545 30541 raft_consensus.cc:357] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER }
I20250113 03:10:26.372593 30541 raft_consensus.cc:383] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:26.372606 30541 raft_consensus.cc:738] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ed2653247a48ae87402267c04c3c96, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.372635 30541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER }
I20250113 03:10:26.372674 30541 raft_consensus.cc:397] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:10:26.372694 30541 raft_consensus.cc:491] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:10:26.372706 30541 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:26.373072 30541 raft_consensus.cc:513] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER }
I20250113 03:10:26.373117 30541 leader_election.cc:304] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ed2653247a48ae87402267c04c3c96; no voters:
I20250113 03:10:26.373188 30541 leader_election.cc:290] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:10:26.373232 30544 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:26.373291 30541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:10:26.373344 30544 raft_consensus.cc:695] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 1 LEADER]: Becoming Leader. State: Replica: 37ed2653247a48ae87402267c04c3c96, State: Running, Role: LEADER
I20250113 03:10:26.373402 30544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER }
I20250113 03:10:26.373562 30545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ed2653247a48ae87402267c04c3c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER } }
I20250113 03:10:26.373616 30545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:26.373582 30546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ed2653247a48ae87402267c04c3c96. Latest consensus state: current_term: 1 leader_uuid: "37ed2653247a48ae87402267c04c3c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ed2653247a48ae87402267c04c3c96" member_type: VOTER } }
I20250113 03:10:26.373665 30546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:26.373940 30550 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:10:26.374047 30550 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:10:26.374373 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:10:26.374478 30550 catalog_manager.cc:1348] Generated new cluster ID: fd5ffd73b09546b4803c7969098d4067
I20250113 03:10:26.374516 30550 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:10:26.391960 30550 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:10:26.392297 30550 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:10:26.395818 30550 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96: Generated new TSK 0
I20250113 03:10:26.395874 30550 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:10:26.405974 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:26.406993 30562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.407516 30565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.408222 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:26.408866 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:26.408926 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:26.408946 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:26.408955 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737826408956 us; error 0 us; skew 500 ppm
I20250113 03:10:26.409387 20553 webserver.cc:458] Webserver started at http://127.20.18.65:44115/ using document root <none> and password file <none>
I20250113 03:10:26.409441 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:26.409467 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:26.409495 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:26.409698 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "741a77efb4bd47b094ab3cc7c665d48a"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.410357 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:26.410688 30570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.410820 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.410863 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "741a77efb4bd47b094ab3cc7c665d48a"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.410907 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:26.415390 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:26.415508 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:26.415603 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:26.415730 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:26.415752 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.415776 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:26.415798 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.418051 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:39105
I20250113 03:10:26.418467 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:10:26.418511 30633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:39105 every 8 connection(s)
W20250113 03:10:26.419505 30636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.420071 30638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:26.420709 20553 server_base.cc:1034] running on GCE node
W20250113 03:10:26.421172 30641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:26.421298 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:26.421329 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:26.421345 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737826421346 us; error 0 us; skew 500 ppm
I20250113 03:10:26.421599 30634 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33433
I20250113 03:10:26.421653 30634 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:26.421737 30634 heartbeater.cc:510] Master 127.20.18.126:33433 requested a full tablet report, sending...
I20250113 03:10:26.421954 30505 ts_manager.cc:194] Registered new tserver with Master: 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105)
I20250113 03:10:26.421962 20553 webserver.cc:458] Webserver started at http://127.20.18.66:40011/ using document root <none> and password file <none>
I20250113 03:10:26.422067 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:26.422106 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:26.422142 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:26.422394 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "643f022f6f4349339b8e470653e4b9dd"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.422663 30505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53948
I20250113 03:10:26.423154 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:26.423484 30646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.423599 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.423633 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "643f022f6f4349339b8e470653e4b9dd"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.423663 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:26.430534 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:26.430686 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:26.430801 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:26.430963 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:26.430989 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.431018 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:26.431046 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.433751 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:32977
I20250113 03:10:26.433773 30709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:32977 every 8 connection(s)
I20250113 03:10:26.434060 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:26.435204 30713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.435784 30716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:26.436412 30714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:26.436749 30710 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33433
I20250113 03:10:26.436791 30710 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:26.436861 30710 heartbeater.cc:510] Master 127.20.18.126:33433 requested a full tablet report, sending...
I20250113 03:10:26.437033 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:26.437062 30505 ts_manager.cc:194] Registered new tserver with Master: 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977)
I20250113 03:10:26.437162 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:26.437194 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:26.437222 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737826437221 us; error 0 us; skew 500 ppm
I20250113 03:10:26.437518 30505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53962
I20250113 03:10:26.437848 20553 webserver.cc:458] Webserver started at http://127.20.18.67:41331/ using document root <none> and password file <none>
I20250113 03:10:26.437937 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:26.437983 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:26.438019 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:26.438213 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "9a835689ed0e4256930059bd12b3c8b3"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.439074 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:26.439453 30721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.439567 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.439606 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "9a835689ed0e4256930059bd12b3c8b3"
format_stamp: "Formatted at 2025-01-13 03:10:26 on dist-test-slave-d4dl"
I20250113 03:10:26.439651 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:26.446529 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:26.446720 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:26.446846 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:26.447007 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:26.447034 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.447062 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:26.447089 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.450098 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:34247
I20250113 03:10:26.450145 30784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:34247 every 8 connection(s)
I20250113 03:10:26.452720 30785 heartbeater.cc:346] Connected to a master server at 127.20.18.126:33433
I20250113 03:10:26.452765 30785 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:26.452872 30785 heartbeater.cc:510] Master 127.20.18.126:33433 requested a full tablet report, sending...
I20250113 03:10:26.453055 30505 ts_manager.cc:194] Registered new tserver with Master: 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:26.453320 30505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53976
I20250113 03:10:26.453518 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003254469s
I20250113 03:10:26.453773 20553 test_util.cc:274] Using random seed: -919160564
I20250113 03:10:26.457351 30505 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53986:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:10:26.457448 30505 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:10:26.460973 30674 tablet_service.cc:1467] Processing CreateTablet for tablet 19342834010a4a2ebccb026f55b7d2a9 (DEFAULT_TABLE table=test-workload [id=15177fe26255410c9a3d61d009ebf2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:26.461079 30674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19342834010a4a2ebccb026f55b7d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:26.461230 30598 tablet_service.cc:1467] Processing CreateTablet for tablet 19342834010a4a2ebccb026f55b7d2a9 (DEFAULT_TABLE table=test-workload [id=15177fe26255410c9a3d61d009ebf2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:26.461305 30598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19342834010a4a2ebccb026f55b7d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:26.462195 30800 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: Bootstrap starting.
I20250113 03:10:26.462211 30749 tablet_service.cc:1467] Processing CreateTablet for tablet 19342834010a4a2ebccb026f55b7d2a9 (DEFAULT_TABLE table=test-workload [id=15177fe26255410c9a3d61d009ebf2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:26.462304 30749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19342834010a4a2ebccb026f55b7d2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:26.462335 30801 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Bootstrap starting.
I20250113 03:10:26.462612 30800 tablet_bootstrap.cc:654] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:26.462743 30801 tablet_bootstrap.cc:654] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:26.463147 30800 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: No bootstrap required, opened a new log
I20250113 03:10:26.463199 30800 ts_tablet_manager.cc:1397] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:26.463222 30801 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: No bootstrap required, opened a new log
I20250113 03:10:26.463294 30801 ts_tablet_manager.cc:1397] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:26.463371 30800 raft_consensus.cc:357] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.463433 30801 raft_consensus.cc:357] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.463467 30800 raft_consensus.cc:383] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:26.463487 30804 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Bootstrap starting.
I20250113 03:10:26.463495 30800 raft_consensus.cc:738] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741a77efb4bd47b094ab3cc7c665d48a, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.463572 30800 consensus_queue.cc:260] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.463711 30800 ts_tablet_manager.cc:1428] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.463773 30634 heartbeater.cc:502] Master 127.20.18.126:33433 was elected leader, sending a full tablet report...
I20250113 03:10:26.463485 30801 raft_consensus.cc:383] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:26.463883 30804 tablet_bootstrap.cc:654] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:26.463889 30801 raft_consensus.cc:738] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643f022f6f4349339b8e470653e4b9dd, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.463999 30801 consensus_queue.cc:260] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.464149 30801 ts_tablet_manager.cc:1428] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250113 03:10:26.464183 30710 heartbeater.cc:502] Master 127.20.18.126:33433 was elected leader, sending a full tablet report...
I20250113 03:10:26.464498 30804 tablet_bootstrap.cc:492] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: No bootstrap required, opened a new log
I20250113 03:10:26.464553 30804 ts_tablet_manager.cc:1397] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:26.464712 30804 raft_consensus.cc:357] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.464763 30804 raft_consensus.cc:383] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:26.464780 30804 raft_consensus.cc:738] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a835689ed0e4256930059bd12b3c8b3, State: Initialized, Role: FOLLOWER
I20250113 03:10:26.464824 30804 consensus_queue.cc:260] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.464958 30804 ts_tablet_manager.cc:1428] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:26.465001 30785 heartbeater.cc:502] Master 127.20.18.126:33433 was elected leader, sending a full tablet report...
I20250113 03:10:26.475939 30808 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:26.475994 30808 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.476120 30808 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105)
I20250113 03:10:26.478003 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd" is_pre_election: true
I20250113 03:10:26.478080 30684 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 0.
I20250113 03:10:26.478142 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a" is_pre_election: true
I20250113 03:10:26.478206 30608 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 0.
I20250113 03:10:26.478216 30724 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3; no voters:
I20250113 03:10:26.478287 30808 raft_consensus.cc:2798] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:10:26.478322 30808 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:26.478338 30808 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:26.478751 30808 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.478825 30808 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 1 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105)
I20250113 03:10:26.478945 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:26.478955 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a"
I20250113 03:10:26.478994 30684 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:26.479027 30608 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:26.479489 30684 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 1.
I20250113 03:10:26.479538 30608 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 1.
I20250113 03:10:26.479619 30724 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3; no voters:
I20250113 03:10:26.479698 30808 raft_consensus.cc:2798] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:26.479802 30808 raft_consensus.cc:695] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 9a835689ed0e4256930059bd12b3c8b3, State: Running, Role: LEADER
I20250113 03:10:26.479851 30808 consensus_queue.cc:237] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:26.480211 30505 catalog_manager.cc:5526] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67). New cstate: current_term: 1 leader_uuid: "9a835689ed0e4256930059bd12b3c8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:26.490901 30608 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 1 FOLLOWER]: Refusing update from remote peer 9a835689ed0e4256930059bd12b3c8b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:10:26.490931 30684 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 1 FOLLOWER]: Refusing update from remote peer 9a835689ed0e4256930059bd12b3c8b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:10:26.491065 30808 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:26.491133 30810 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:36.541473 30856 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a835689ed0e4256930059bd12b3c8b3)
I20250113 03:10:36.541535 30856 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:36.542100 30856 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.542230 30856 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 2 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.542955 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 1. Current term is 2. Ops: []
I20250113 03:10:36.543131 30835 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 1.16649, Next index: 16650, Last known committed idx: 16649, Time since last communication: 0.000s
I20250113 03:10:36.544788 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16649 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.544816 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16649 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.545002 30573 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 741a77efb4bd47b094ab3cc7c665d48a; no voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.545078 30856 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 03:10:36.552146 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 1. Current term is 2. Ops: [1.16650-1.16650]
I20250113 03:10:36.552330 30847 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 1.16649, Next index: 16650, Last known committed idx: 16649, Time since last communication: 0.000s
I20250113 03:10:36.576260 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 1. Current term is 2. Ops: [1.16650-1.16650]
I20250113 03:10:36.576449 30843 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 1.16649, Next index: 16650, Last known committed idx: 16649, Time since last communication: 0.000s
I20250113 03:10:36.578301 30835 raft_consensus.cc:3049] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 03:10:36.578336 30835 raft_consensus.cc:738] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a835689ed0e4256930059bd12b3c8b3, State: Running, Role: LEADER
I20250113 03:10:36.578372 30835 consensus_queue.cc:260] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16650, Committed index: 16650, Last appended: 1.16650, Last appended by leader: 16650, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.578431 30835 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:36.580442 30856 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:36.580489 30856 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:36.580930 30856 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.581022 30856 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 3 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.581187 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 3 candidate_status { last_received { term: 1 index: 16649 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.581219 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 3 candidate_status { last_received { term: 1 index: 16649 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.581290 30759 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:36.581857 30759 raft_consensus.cc:2405] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 741a77efb4bd47b094ab3cc7c665d48a for term 3 because replica has last-logged OpId of term: 1 index: 16650, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 16649.
I20250113 03:10:36.582013 30573 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 741a77efb4bd47b094ab3cc7c665d48a; no voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.582093 30856 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250113 03:10:36.583066 30862 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a835689ed0e4256930059bd12b3c8b3)
I20250113 03:10:36.583098 30862 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 1 FOLLOWER]: Advancing to term 2
I20250113 03:10:36.583539 30862 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.583678 30862 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 2 election: Requested vote from peers 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.585846 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 16650 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.585902 30759 raft_consensus.cc:2363] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd for earlier term 2. Current term is 3.
I20250113 03:10:36.585911 30607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 16650 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a"
I20250113 03:10:36.585994 30607 raft_consensus.cc:2363] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd for earlier term 2. Current term is 3.
I20250113 03:10:36.586045 30649 leader_election.cc:400] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 2 election: Vote denied by peer 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247) with higher term. Message: Invalid argument: T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd for earlier term 2. Current term is 3.
I20250113 03:10:36.586084 30649 leader_election.cc:403] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 03:10:36.586141 30649 leader_election.cc:400] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 2 election: Vote denied by peer 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105) with higher term. Message: Invalid argument: T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd for earlier term 2. Current term is 3.
I20250113 03:10:36.586143 30862 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 2 FOLLOWER]: Advancing to term 3
I20250113 03:10:36.586575 30862 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247) with higher term. Message: Invalid argument: T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd for earlier term 2. Current term is 3.
I20250113 03:10:36.607681 30843 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:36.607722 30843 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:10:36.608121 30843 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.608201 30843 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 4 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105)
I20250113 03:10:36.608332 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 16650 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.608356 30607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 16650 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a"
I20250113 03:10:36.608379 30684 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:10:36.608430 30607 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 3 FOLLOWER]: Advancing to term 4
I20250113 03:10:36.608986 30684 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 4.
I20250113 03:10:36.609023 30607 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a835689ed0e4256930059bd12b3c8b3 in term 4.
I20250113 03:10:36.609153 30724 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3; no voters:
I20250113 03:10:36.609229 30843 raft_consensus.cc:2798] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 FOLLOWER]: Leader election won for term 4
I20250113 03:10:36.609283 30843 raft_consensus.cc:695] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 LEADER]: Becoming Leader. State: Replica: 9a835689ed0e4256930059bd12b3c8b3, State: Running, Role: LEADER
I20250113 03:10:36.609329 30843 consensus_queue.cc:237] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16649, Committed index: 16649, Last appended: 1.16650, Last appended by leader: 16650, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.609719 30505 catalog_manager.cc:5526] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "9a835689ed0e4256930059bd12b3c8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } health_report { overall_health: HEALTHY } } }
W20250113 03:10:36.622069 30835 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
I20250113 03:10:36.625404 30607 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 4 FOLLOWER]: Refusing update from remote peer 9a835689ed0e4256930059bd12b3c8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 16649. Preceding OpId from leader: term: 4 index: 16651. (index mismatch)
I20250113 03:10:36.625417 30684 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 4 FOLLOWER]: Refusing update from remote peer 9a835689ed0e4256930059bd12b3c8b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 16650. Preceding OpId from leader: term: 4 index: 16651. (index mismatch)
I20250113 03:10:36.625571 30847 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16651, Last known committed idx: 16649, Time since last communication: 0.000s
I20250113 03:10:36.625766 30847 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16651, Last known committed idx: 16650, Time since last communication: 0.000s
W20250113 03:10:36.659319 30843 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
I20250113 03:10:36.740257 30856 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9a835689ed0e4256930059bd12b3c8b3)
I20250113 03:10:36.740304 30856 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 4 FOLLOWER]: Advancing to term 5
I20250113 03:10:36.740845 30856 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.740959 30856 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 5 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.740959 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:36.741086 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.741130 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.741242 30573 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 741a77efb4bd47b094ab3cc7c665d48a; no voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.741238 30808 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.741358 30856 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 03:10:36.768085 30862 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9a835689ed0e4256930059bd12b3c8b3)
I20250113 03:10:36.768133 30862 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 4 FOLLOWER]: Advancing to term 5
I20250113 03:10:36.768635 30862 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.768728 30862 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 5 election: Requested vote from peers 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.768852 30607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a"
I20250113 03:10:36.768888 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 5 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.768927 30607 raft_consensus.cc:2388] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 643f022f6f4349339b8e470653e4b9dd in current term 5: Already voted for candidate 741a77efb4bd47b094ab3cc7c665d48a in this term.
I20250113 03:10:36.769039 30649 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd; no voters: 741a77efb4bd47b094ab3cc7c665d48a, 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.769105 30862 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 03:10:36.771567 30684 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 5 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:36.771697 30835 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.773939 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 5. Ops: []
I20250113 03:10:36.774092 30808 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.777570 30684 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 5 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 5. Ops: [4.16654-4.16654]
I20250113 03:10:36.777590 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 5. Ops: [4.16654-4.16654]
I20250113 03:10:36.777714 30835 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.777813 30847 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.778605 30856 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:36.778640 30856 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 5 FOLLOWER]: Advancing to term 6
I20250113 03:10:36.779034 30856 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.779114 30856 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 6 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.779263 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 6 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.779278 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "741a77efb4bd47b094ab3cc7c665d48a" candidate_term: 6 candidate_status { last_received { term: 4 index: 16653 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.779322 30684 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 5 FOLLOWER]: Advancing to term 6
I20250113 03:10:36.779779 30684 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741a77efb4bd47b094ab3cc7c665d48a in term 6.
I20250113 03:10:36.779902 30573 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd, 741a77efb4bd47b094ab3cc7c665d48a; no voters: 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.779976 30856 raft_consensus.cc:2798] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 FOLLOWER]: Leader election won for term 6
I20250113 03:10:36.780058 30856 raft_consensus.cc:695] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 LEADER]: Becoming Leader. State: Replica: 741a77efb4bd47b094ab3cc7c665d48a, State: Running, Role: LEADER
I20250113 03:10:36.780094 30856 consensus_queue.cc:237] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16653, Committed index: 16653, Last appended: 4.16653, Last appended by leader: 16653, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.780431 30504 catalog_manager.cc:5526] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a reported cstate change: term changed from 4 to 6, leader changed from 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67) to 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65). New cstate: current_term: 6 leader_uuid: "741a77efb4bd47b094ab3cc7c665d48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:36.802173 30607 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 LEADER]: Rejecting Update request from peer 9a835689ed0e4256930059bd12b3c8b3 for earlier term 4. Current term is 6. Ops: [4.16654-4.16654]
I20250113 03:10:36.802337 30847 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 }, Status: INVALID_TERM, Last received: 4.16653, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.802419 30847 consensus_queue.cc:1225] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [LEADER]: Peer 741a77efb4bd47b094ab3cc7c665d48a log is divergent from this leader: its last log entry 6.16654 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16653
I20250113 03:10:36.802475 30684 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 6 FOLLOWER]: Refusing update from remote peer 741a77efb4bd47b094ab3cc7c665d48a: Log matching property violated. Preceding OpId in replica: term: 4 index: 16653. Preceding OpId from leader: term: 6 index: 16654. (index mismatch)
I20250113 03:10:36.802637 30856 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
I20250113 03:10:36.804392 30759 raft_consensus.cc:3049] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 LEADER]: Stepping down as leader of term 4
I20250113 03:10:36.804427 30759 raft_consensus.cc:738] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9a835689ed0e4256930059bd12b3c8b3, State: Running, Role: LEADER
I20250113 03:10:36.804459 30759 consensus_queue.cc:260] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16653, Committed index: 16653, Last appended: 4.16654, Last appended by leader: 16654, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.804562 30759 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 4 FOLLOWER]: Advancing to term 6
I20250113 03:10:36.805006 30759 raft_consensus.cc:1270] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 6 FOLLOWER]: Refusing update from remote peer 741a77efb4bd47b094ab3cc7c665d48a: Log matching property violated. Preceding OpId in replica: term: 4 index: 16654. Preceding OpId from leader: term: 6 index: 16654. (term mismatch)
I20250113 03:10:36.805037 30759 pending_rounds.cc:77] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Aborting all ops after (but not including) 16653
I20250113 03:10:36.805048 30759 pending_rounds.cc:99] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 4.16654
W20250113 03:10:36.805068 30759 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35614: Aborted: Op aborted by new leader
I20250113 03:10:36.805330 30871 consensus_queue.cc:1035] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16654, Last known committed idx: 16653, Time since last communication: 0.000s
W20250113 03:10:36.808140 30674 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica 643f022f6f4349339b8e470653e4b9dd is not leader of this config: current role FOLLOWER
I20250113 03:10:36.887939 30862 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 741a77efb4bd47b094ab3cc7c665d48a)
I20250113 03:10:36.887984 30862 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 6 FOLLOWER]: Advancing to term 7
I20250113 03:10:36.888581 30862 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.888660 30684 raft_consensus.cc:1235] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Rejecting Update request from peer 741a77efb4bd47b094ab3cc7c665d48a for earlier term 6. Current term is 7. Ops: []
I20250113 03:10:36.888690 30862 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 7 election: Requested vote from peers 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.888870 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 16657 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.888828 30607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 7 candidate_status { last_received { term: 6 index: 16657 } } ignore_live_leader: false dest_uuid: "741a77efb4bd47b094ab3cc7c665d48a"
I20250113 03:10:36.888939 30856 consensus_queue.cc:1046] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 }, Status: INVALID_TERM, Last received: 6.16657, Next index: 16658, Last known committed idx: 16657, Time since last communication: 0.000s
I20250113 03:10:36.889077 30649 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd; no voters: 741a77efb4bd47b094ab3cc7c665d48a, 9a835689ed0e4256930059bd12b3c8b3
I20250113 03:10:36.889168 30862 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250113 03:10:36.889811 20553 tablet_server.cc:178] TabletServer@127.20.18.65:0 shutting down...
I20250113 03:10:36.891892 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:36.892004 20553 tablet_replica.cc:331] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a: stopping tablet replica
I20250113 03:10:36.892035 20553 raft_consensus.cc:2238] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 LEADER]: Raft consensus shutting down.
I20250113 03:10:36.918691 30847 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 741a77efb4bd47b094ab3cc7c665d48a)
I20250113 03:10:36.918733 30847 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 6 FOLLOWER]: Advancing to term 7
I20250113 03:10:36.919164 30847 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.919251 30847 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 7 election: Requested vote from peers 643f022f6f4349339b8e470653e4b9dd (127.20.18.66:32977), 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105)
I20250113 03:10:36.919399 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "9a835689ed0e4256930059bd12b3c8b3" candidate_term: 7 candidate_status { last_received { term: 6 index: 16657 } } ignore_live_leader: false dest_uuid: "643f022f6f4349339b8e470653e4b9dd"
I20250113 03:10:36.919445 30684 raft_consensus.cc:2388] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a835689ed0e4256930059bd12b3c8b3 in current term 7: Already voted for candidate 643f022f6f4349339b8e470653e4b9dd in this term.
W20250113 03:10:36.919446 30724 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 19 similar messages]
W20250113 03:10:36.919823 30724 leader_election.cc:336] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250113 03:10:36.919865 30724 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a835689ed0e4256930059bd12b3c8b3; no voters: 643f022f6f4349339b8e470653e4b9dd, 741a77efb4bd47b094ab3cc7c665d48a
I20250113 03:10:36.919924 30847 raft_consensus.cc:2743] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250113 03:10:36.921576 30871 raft_consensus.cc:927] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 LEADER]: Unable to handle notification of new term (7): Illegal state: RaftConsensus is not running: State = Stopping
I20250113 03:10:36.921650 20553 raft_consensus.cc:2267] T 19342834010a4a2ebccb026f55b7d2a9 P 741a77efb4bd47b094ab3cc7c665d48a [term 6 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:36.931507 30862 raft_consensus.cc:491] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:36.931563 30862 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 7 FOLLOWER]: Advancing to term 8
I20250113 03:10:36.931951 30862 raft_consensus.cc:513] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.932044 30862 leader_election.cc:290] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 8 election: Requested vote from peers 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105), 9a835689ed0e4256930059bd12b3c8b3 (127.20.18.67:34247)
I20250113 03:10:36.932237 30759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19342834010a4a2ebccb026f55b7d2a9" candidate_uuid: "643f022f6f4349339b8e470653e4b9dd" candidate_term: 8 candidate_status { last_received { term: 6 index: 16657 } } ignore_live_leader: false dest_uuid: "9a835689ed0e4256930059bd12b3c8b3"
I20250113 03:10:36.932283 30759 raft_consensus.cc:3054] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 7 FOLLOWER]: Advancing to term 8
W20250113 03:10:36.932483 30649 leader_election.cc:336] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65:39105): Network error: Client connection negotiation failed: client connection to 127.20.18.65:39105: connect: Connection refused (error 111)
I20250113 03:10:36.932735 30759 raft_consensus.cc:2463] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 643f022f6f4349339b8e470653e4b9dd in term 8.
I20250113 03:10:36.932870 30649 leader_election.cc:304] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 643f022f6f4349339b8e470653e4b9dd, 9a835689ed0e4256930059bd12b3c8b3; no voters: 741a77efb4bd47b094ab3cc7c665d48a
I20250113 03:10:36.932940 30862 raft_consensus.cc:2798] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 FOLLOWER]: Leader election won for term 8
I20250113 03:10:36.933013 30862 raft_consensus.cc:695] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 LEADER]: Becoming Leader. State: Replica: 643f022f6f4349339b8e470653e4b9dd, State: Running, Role: LEADER
I20250113 03:10:36.933043 30862 consensus_queue.cc:237] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16657, Committed index: 16657, Last appended: 6.16657, Last appended by leader: 16657, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } }
I20250113 03:10:36.933346 30504 catalog_manager.cc:5526] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd reported cstate change: term changed from 6 to 8, leader changed from 741a77efb4bd47b094ab3cc7c665d48a (127.20.18.65) to 643f022f6f4349339b8e470653e4b9dd (127.20.18.66). New cstate: current_term: 8 leader_uuid: "643f022f6f4349339b8e470653e4b9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643f022f6f4349339b8e470653e4b9dd" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 32977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "741a77efb4bd47b094ab3cc7c665d48a" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a835689ed0e4256930059bd12b3c8b3" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250113 03:10:36.937340 20553 tablet_server.cc:195] TabletServer@127.20.18.65:0 shutdown complete.
I20250113 03:10:36.937999 20553 tablet_server.cc:178] TabletServer@127.20.18.66:0 shutting down...
I20250113 03:10:36.946103 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:36.946214 20553 tablet_replica.cc:331] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: stopping tablet replica
I20250113 03:10:36.946250 20553 raft_consensus.cc:2238] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 LEADER]: Raft consensus shutting down.
I20250113 03:10:36.946292 20553 pending_rounds.cc:62] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Trying to abort 1 pending ops.
I20250113 03:10:36.946313 20553 pending_rounds.cc:69] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd: Aborting op as it isn't in flight: id { term: 8 index: 16658 } timestamp: 7113678180078137344 op_type: NO_OP noop_request { }
I20250113 03:10:36.946350 20553 raft_consensus.cc:2883] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250113 03:10:36.946377 20553 raft_consensus.cc:2267] T 19342834010a4a2ebccb026f55b7d2a9 P 643f022f6f4349339b8e470653e4b9dd [term 8 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:36.968254 20553 tablet_server.cc:195] TabletServer@127.20.18.66:0 shutdown complete.
I20250113 03:10:36.968859 20553 tablet_server.cc:178] TabletServer@127.20.18.67:0 shutting down...
I20250113 03:10:36.971215 20553 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250113 03:10:36.971311 20553 tablet_replica.cc:331] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3: stopping tablet replica
I20250113 03:10:36.971372 20553 raft_consensus.cc:2238] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250113 03:10:36.971405 20553 raft_consensus.cc:2267] T 19342834010a4a2ebccb026f55b7d2a9 P 9a835689ed0e4256930059bd12b3c8b3 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:36.975015 20553 tablet_server.cc:195] TabletServer@127.20.18.67:0 shutdown complete.
I20250113 03:10:36.975687 20553 master.cc:537] Master@127.20.18.126:33433 shutting down...
I20250113 03:10:36.977411 20553 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 1 LEADER]: Raft consensus shutting down.
I20250113 03:10:36.977455 20553 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 03:10:36.977470 20553 tablet_replica.cc:331] T 00000000000000000000000000000000 P 37ed2653247a48ae87402267c04c3c96: stopping tablet replica
I20250113 03:10:36.988720 20553 master.cc:559] Master@127.20.18.126:33433 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm (10658 ms)
[----------] 2 tests from TxnParticipantElectionStormITest (27532 ms total)
[----------] 2 tests from ShouldGCWals/ParticipantCopyITest
[ RUN ] ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/0
I20250113 03:10:37.007400 20553 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.18.126:40915
I20250113 03:10:37.007509 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:37.008603 30875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.009285 30878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.009837 30876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:37.010557 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:37.010715 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:37.010751 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:37.010763 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737837010764 us; error 0 us; skew 500 ppm
I20250113 03:10:37.011312 20553 webserver.cc:458] Webserver started at http://127.20.18.126:42101/ using document root <none> and password file <none>
I20250113 03:10:37.011384 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:37.011443 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:37.011525 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:37.011741 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/master-0-root/instance:
uuid: "6b45c552a0d3422ba326065fa2203da9"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.012473 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.012902 30883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.013000 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.013041 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/master-0-root
uuid: "6b45c552a0d3422ba326065fa2203da9"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.013074 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:37.040721 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:37.040853 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:37.042919 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.126:40915
I20250113 03:10:37.048216 30936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:37.048694 30935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.126:40915 every 8 connection(s)
I20250113 03:10:37.054095 30936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9: Bootstrap starting.
I20250113 03:10:37.054342 30936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:37.054725 30936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9: No bootstrap required, opened a new log
I20250113 03:10:37.054821 30936 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER }
I20250113 03:10:37.054857 30936 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:37.054875 30936 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b45c552a0d3422ba326065fa2203da9, State: Initialized, Role: FOLLOWER
I20250113 03:10:37.054919 30936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER }
I20250113 03:10:37.054950 30936 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 03:10:37.054968 30936 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 03:10:37.054993 30936 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:37.055361 30936 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER }
I20250113 03:10:37.055408 30936 leader_election.cc:304] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b45c552a0d3422ba326065fa2203da9; no voters:
I20250113 03:10:37.055498 30936 leader_election.cc:290] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 03:10:37.055534 30939 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:37.055608 30936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 03:10:37.055645 30939 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [term 1 LEADER]: Becoming Leader. State: Replica: 6b45c552a0d3422ba326065fa2203da9, State: Running, Role: LEADER
I20250113 03:10:37.055704 30939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER }
I20250113 03:10:37.055891 30940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b45c552a0d3422ba326065fa2203da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER } }
I20250113 03:10:37.055984 30940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:37.055905 30941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b45c552a0d3422ba326065fa2203da9. Latest consensus state: current_term: 1 leader_uuid: "6b45c552a0d3422ba326065fa2203da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b45c552a0d3422ba326065fa2203da9" member_type: VOTER } }
I20250113 03:10:37.056155 30941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9 [sys.catalog]: This master's current role is: LEADER
I20250113 03:10:37.056233 30946 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 03:10:37.056352 30946 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 03:10:37.056803 20553 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250113 03:10:37.056825 30946 catalog_manager.cc:1348] Generated new cluster ID: 8f6df9d4a4c04b3c9d279be0831ebb89
I20250113 03:10:37.056861 30946 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 03:10:37.076022 30946 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 03:10:37.076352 30946 catalog_manager.cc:1505] Loading token signing keys...
I20250113 03:10:37.079628 30946 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6b45c552a0d3422ba326065fa2203da9: Generated new TSK 0
I20250113 03:10:37.079694 30946 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 03:10:37.088371 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 03:10:37.089778 30960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.089951 30957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.090668 30958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:37.091287 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:37.091347 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:37.091373 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:37.091398 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737837091398 us; error 0 us; skew 500 ppm
I20250113 03:10:37.091909 20553 webserver.cc:458] Webserver started at http://127.20.18.65:44085/ using document root <none> and password file <none>
I20250113 03:10:37.091969 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:37.092006 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:37.092046 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:37.092238 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/instance:
uuid: "80237f8c3f2c432facd353586e5b82d8"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.092996 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250113 03:10:37.093358 30965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.093475 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.093513 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root
uuid: "80237f8c3f2c432facd353586e5b82d8"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.093552 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:37.112007 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:37.112138 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:37.112252 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:37.112398 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:37.112421 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.112448 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:37.112475 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.114881 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.65:43633
I20250113 03:10:37.115334 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:10:37.115535 31028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.65:43633 every 8 connection(s)
W20250113 03:10:37.116801 31032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.116966 31031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.117587 31034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:37.118072 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:37.118173 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:37.118212 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:37.118229 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737837118229 us; error 0 us; skew 500 ppm
I20250113 03:10:37.118608 31029 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40915
I20250113 03:10:37.118656 31029 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:37.118748 31029 heartbeater.cc:510] Master 127.20.18.126:40915 requested a full tablet report, sending...
I20250113 03:10:37.118855 20553 webserver.cc:458] Webserver started at http://127.20.18.66:45951/ using document root <none> and password file <none>
I20250113 03:10:37.118928 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:37.118959 30900 ts_manager.cc:194] Registered new tserver with Master: 80237f8c3f2c432facd353586e5b82d8 (127.20.18.65:43633)
I20250113 03:10:37.118969 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:37.119031 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:37.119235 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/instance:
uuid: "992c4345dfd4499390689246ae3d3cd9"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.119422 30900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52982
I20250113 03:10:37.120007 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.120345 31041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.120442 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.120476 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root
uuid: "992c4345dfd4499390689246ae3d3cd9"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.120513 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:37.127784 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:37.127903 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:37.128001 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:37.128134 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:37.128154 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.128170 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:37.128192 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.130508 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.66:43861
I20250113 03:10:37.130980 20553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 03:10:37.131120 31104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.66:43861 every 8 connection(s)
W20250113 03:10:37.131922 31107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.132419 31111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 03:10:37.133009 31108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 03:10:37.133200 20553 server_base.cc:1034] running on GCE node
I20250113 03:10:37.133278 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250113 03:10:37.133311 20553 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250113 03:10:37.133322 20553 hybrid_clock.cc:648] HybridClock initialized: now 1736737837133321 us; error 0 us; skew 500 ppm
I20250113 03:10:37.133846 31105 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40915
I20250113 03:10:37.133890 20553 webserver.cc:458] Webserver started at http://127.20.18.67:43241/ using document root <none> and password file <none>
I20250113 03:10:37.133898 31105 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:37.133978 20553 fs_manager.cc:362] Metadata directory not provided
I20250113 03:10:37.134006 31105 heartbeater.cc:510] Master 127.20.18.126:40915 requested a full tablet report, sending...
I20250113 03:10:37.134017 20553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 03:10:37.134078 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 03:10:37.134195 30900 ts_manager.cc:194] Registered new tserver with Master: 992c4345dfd4499390689246ae3d3cd9 (127.20.18.66:43861)
I20250113 03:10:37.134305 20553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/instance:
uuid: "8e6ae3b64a874690b4c926cbede85053"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.134647 30900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52986
I20250113 03:10:37.135177 20553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.135555 31116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.135665 20553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.135702 20553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root
uuid: "8e6ae3b64a874690b4c926cbede85053"
format_stamp: "Formatted at 2025-01-13 03:10:37 on dist-test-slave-d4dl"
I20250113 03:10:37.135742 20553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 03:10:37.145035 20553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 03:10:37.145176 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 03:10:37.145289 20553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 03:10:37.145427 20553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 03:10:37.145452 20553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.145474 20553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 03:10:37.145503 20553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.148123 20553 rpc_server.cc:307] RPC server started. Bound to: 127.20.18.67:36743
I20250113 03:10:37.148146 31179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.18.67:36743 every 8 connection(s)
I20250113 03:10:37.150378 31180 heartbeater.cc:346] Connected to a master server at 127.20.18.126:40915
I20250113 03:10:37.150424 31180 heartbeater.cc:463] Registering TS with master...
I20250113 03:10:37.150491 31180 heartbeater.cc:510] Master 127.20.18.126:40915 requested a full tablet report, sending...
I20250113 03:10:37.150667 30900 ts_manager.cc:194] Registered new tserver with Master: 8e6ae3b64a874690b4c926cbede85053 (127.20.18.67:36743)
I20250113 03:10:37.150945 30900 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52994
I20250113 03:10:37.151531 20553 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003280609s
I20250113 03:10:37.151846 20553 test_util.cc:274] Using random seed: -908462490
I20250113 03:10:37.155474 30900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53006:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 03:10:37.155575 30900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 03:10:37.159163 30993 tablet_service.cc:1467] Processing CreateTablet for tablet bee32ae0dcbc4912aeb5d6155f3158bb (DEFAULT_TABLE table=test-workload [id=a68c202659494d719e4a3e3af7c407a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:37.159263 30993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee32ae0dcbc4912aeb5d6155f3158bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:37.159440 31144 tablet_service.cc:1467] Processing CreateTablet for tablet bee32ae0dcbc4912aeb5d6155f3158bb (DEFAULT_TABLE table=test-workload [id=a68c202659494d719e4a3e3af7c407a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:37.159528 31144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee32ae0dcbc4912aeb5d6155f3158bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:37.159925 31069 tablet_service.cc:1467] Processing CreateTablet for tablet bee32ae0dcbc4912aeb5d6155f3158bb (DEFAULT_TABLE table=test-workload [id=a68c202659494d719e4a3e3af7c407a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 03:10:37.160013 31069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee32ae0dcbc4912aeb5d6155f3158bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 03:10:37.160658 31195 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Bootstrap starting.
I20250113 03:10:37.160771 31196 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Bootstrap starting.
I20250113 03:10:37.161139 31197 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Bootstrap starting.
I20250113 03:10:37.161144 31196 tablet_bootstrap.cc:654] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:37.161229 31195 tablet_bootstrap.cc:654] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:37.161705 31197 tablet_bootstrap.cc:654] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Neither blocks nor log segments found. Creating new log.
I20250113 03:10:37.161887 31195 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: No bootstrap required, opened a new log
I20250113 03:10:37.161896 31196 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: No bootstrap required, opened a new log
I20250113 03:10:37.161944 31195 ts_tablet_manager.cc:1397] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.161952 31196 ts_tablet_manager.cc:1397] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.162106 31195 raft_consensus.cc:357] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.162151 31196 raft_consensus.cc:357] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.162175 31195 raft_consensus.cc:383] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:37.162214 31196 raft_consensus.cc:383] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:37.162223 31195 raft_consensus.cc:738] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80237f8c3f2c432facd353586e5b82d8, State: Initialized, Role: FOLLOWER
I20250113 03:10:37.162230 31196 raft_consensus.cc:738] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6ae3b64a874690b4c926cbede85053, State: Initialized, Role: FOLLOWER
I20250113 03:10:37.162297 31195 consensus_queue.cc:260] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.162300 31196 consensus_queue.cc:260] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.162510 31029 heartbeater.cc:502] Master 127.20.18.126:40915 was elected leader, sending a full tablet report...
I20250113 03:10:37.162557 31197 tablet_bootstrap.cc:492] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: No bootstrap required, opened a new log
I20250113 03:10:37.162480 31196 ts_tablet_manager.cc:1428] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250113 03:10:37.162612 31197 ts_tablet_manager.cc:1397] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250113 03:10:37.162562 31180 heartbeater.cc:502] Master 127.20.18.126:40915 was elected leader, sending a full tablet report...
I20250113 03:10:37.162744 31197 raft_consensus.cc:357] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.162804 31197 raft_consensus.cc:383] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 03:10:37.162828 31197 raft_consensus.cc:738] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c4345dfd4499390689246ae3d3cd9, State: Initialized, Role: FOLLOWER
I20250113 03:10:37.162863 31197 consensus_queue.cc:260] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.163004 31197 ts_tablet_manager.cc:1428] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250113 03:10:37.163031 31105 heartbeater.cc:502] Master 127.20.18.126:40915 was elected leader, sending a full tablet report...
I20250113 03:10:37.162480 31195 ts_tablet_manager.cc:1428] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20250113 03:10:37.165980 31030 tablet.cc:2367] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 03:10:37.228838 31203 raft_consensus.cc:491] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 03:10:37.228888 31203 raft_consensus.cc:513] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.229017 31203 leader_election.cc:290] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e6ae3b64a874690b4c926cbede85053 (127.20.18.67:36743), 80237f8c3f2c432facd353586e5b82d8 (127.20.18.65:43633)
I20250113 03:10:37.231618 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee32ae0dcbc4912aeb5d6155f3158bb" candidate_uuid: "992c4345dfd4499390689246ae3d3cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6ae3b64a874690b4c926cbede85053" is_pre_election: true
I20250113 03:10:37.231676 31154 raft_consensus.cc:2463] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992c4345dfd4499390689246ae3d3cd9 in term 0.
I20250113 03:10:37.231657 31003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee32ae0dcbc4912aeb5d6155f3158bb" candidate_uuid: "992c4345dfd4499390689246ae3d3cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80237f8c3f2c432facd353586e5b82d8" is_pre_election: true
I20250113 03:10:37.231738 31003 raft_consensus.cc:2463] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992c4345dfd4499390689246ae3d3cd9 in term 0.
I20250113 03:10:37.231854 31043 leader_election.cc:304] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e6ae3b64a874690b4c926cbede85053, 992c4345dfd4499390689246ae3d3cd9; no voters:
I20250113 03:10:37.231957 31203 raft_consensus.cc:2798] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 03:10:37.232000 31203 raft_consensus.cc:491] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 03:10:37.232021 31203 raft_consensus.cc:3054] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:37.232466 31203 raft_consensus.cc:513] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.232595 31203 leader_election.cc:290] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [CANDIDATE]: Term 1 election: Requested vote from peers 8e6ae3b64a874690b4c926cbede85053 (127.20.18.67:36743), 80237f8c3f2c432facd353586e5b82d8 (127.20.18.65:43633)
I20250113 03:10:37.232712 31154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee32ae0dcbc4912aeb5d6155f3158bb" candidate_uuid: "992c4345dfd4499390689246ae3d3cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6ae3b64a874690b4c926cbede85053"
I20250113 03:10:37.232713 31003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee32ae0dcbc4912aeb5d6155f3158bb" candidate_uuid: "992c4345dfd4499390689246ae3d3cd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80237f8c3f2c432facd353586e5b82d8"
I20250113 03:10:37.232803 31154 raft_consensus.cc:3054] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:37.232825 31003 raft_consensus.cc:3054] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 0 FOLLOWER]: Advancing to term 1
I20250113 03:10:37.233366 31154 raft_consensus.cc:2463] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992c4345dfd4499390689246ae3d3cd9 in term 1.
I20250113 03:10:37.233366 31003 raft_consensus.cc:2463] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992c4345dfd4499390689246ae3d3cd9 in term 1.
I20250113 03:10:37.233517 31043 leader_election.cc:304] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e6ae3b64a874690b4c926cbede85053, 992c4345dfd4499390689246ae3d3cd9; no voters:
I20250113 03:10:37.233608 31203 raft_consensus.cc:2798] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 03:10:37.233711 31203 raft_consensus.cc:695] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [term 1 LEADER]: Becoming Leader. State: Replica: 992c4345dfd4499390689246ae3d3cd9, State: Running, Role: LEADER
I20250113 03:10:37.233754 31203 consensus_queue.cc:237] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } }
I20250113 03:10:37.234156 30898 catalog_manager.cc:5526] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 992c4345dfd4499390689246ae3d3cd9 (127.20.18.66). New cstate: current_term: 1 leader_uuid: "992c4345dfd4499390689246ae3d3cd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "992c4345dfd4499390689246ae3d3cd9" member_type: VOTER last_known_addr { host: "127.20.18.66" port: 43861 } health_report { overall_health: HEALTHY } } }
I20250113 03:10:37.254220 31003 raft_consensus.cc:1270] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8 [term 1 FOLLOWER]: Refusing update from remote peer 992c4345dfd4499390689246ae3d3cd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:10:37.254231 31154 raft_consensus.cc:1270] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053 [term 1 FOLLOWER]: Refusing update from remote peer 992c4345dfd4499390689246ae3d3cd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 03:10:37.254393 31205 consensus_queue.cc:1035] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80237f8c3f2c432facd353586e5b82d8" member_type: VOTER last_known_addr { host: "127.20.18.65" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:37.254453 31203 consensus_queue.cc:1035] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e6ae3b64a874690b4c926cbede85053" member_type: VOTER last_known_addr { host: "127.20.18.67" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 03:10:37.368652 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.370784 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.374023 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.393234 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.012s sys 0.000s Metrics: {"bytes_written":279160,"cfile_init":1,"dirs.queue_time_us":4122,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":540,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:37.393378 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89473 bytes on disk
I20250113 03:10:37.393530 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.393730 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.025s user 0.013s sys 0.000s Metrics: {"bytes_written":277673,"cfile_init":1,"compiler_manager_pool.queue_time_us":5585,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":826,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2361,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9650,"thread_start_us":68,"threads_started":1}
I20250113 03:10:37.393949 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89002 bytes on disk
I20250113 03:10:37.394063 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.395385 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.013s sys 0.000s Metrics: {"bytes_written":279160,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":599,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:37.395502 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89473 bytes on disk
I20250113 03:10:37.395684 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.525099 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.525696 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.529942 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.537892 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.009s sys 0.000s Metrics: {"bytes_written":281553,"cfile_init":1,"dirs.queue_time_us":1319,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":852,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:37.538022 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 90217 bytes on disk
I20250113 03:10:37.538131 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.538223 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014703
I20250113 03:10:37.539238 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":285522,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":600,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9950}
I20250113 03:10:37.539440 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014703
I20250113 03:10:37.556996 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.027s user 0.006s sys 0.005s Metrics: {"bytes_written":286881,"cfile_init":1,"dirs.queue_time_us":2042,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":567,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20250113 03:10:37.557153 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014702
I20250113 03:10:37.588915 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.051s user 0.018s sys 0.008s Metrics: {"bytes_written":556705,"cfile_cache_hit":10,"cfile_cache_hit_bytes":367583,"cfile_cache_miss":42,"cfile_cache_miss_bytes":1012799,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":258,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":503,"drs_written":1,"lbm_read_time_us":616,"lbm_reads_lt_1ms":62,"lbm_write_time_us":1803,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":292259,"rows_written":19500,"thread_start_us":213,"threads_started":5}
I20250113 03:10:37.589244 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 185133 bytes on disk
I20250113 03:10:37.589512 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.597674 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.053s user 0.020s sys 0.004s Metrics: {"bytes_written":559307,"cfile_cache_hit":10,"cfile_cache_hit_bytes":369513,"cfile_cache_miss":42,"cfile_cache_miss_bytes":1017715,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":561,"drs_written":1,"lbm_read_time_us":650,"lbm_reads_lt_1ms":66,"lbm_write_time_us":977,"lbm_writes_lt_1ms":44,"num_input_rowsets":2,"peak_mem_usage":293779,"rows_written":19600,"thread_start_us":239,"threads_started":5}
I20250113 03:10:37.597877 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 186235 bytes on disk
I20250113 03:10:37.598320 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.606982 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.050s user 0.018s sys 0.006s Metrics: {"bytes_written":561947,"cfile_cache_hit":10,"cfile_cache_hit_bytes":371433,"cfile_cache_miss":42,"cfile_cache_miss_bytes":1023397,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":65,"dirs.run_wall_time_us":475,"drs_written":1,"lbm_read_time_us":1976,"lbm_reads_lt_1ms":66,"lbm_write_time_us":2893,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":295309,"rows_written":19700,"thread_start_us":239,"threads_started":5}
I20250113 03:10:37.607259 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 187362 bytes on disk
I20250113 03:10:37.607379 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.680706 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.688309 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.689312 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.694205 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.009s sys 0.000s Metrics: {"bytes_written":271152,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":598,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20250113 03:10:37.694386 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014608
I20250113 03:10:37.701139 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.005s sys 0.005s Metrics: {"bytes_written":281506,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":68,"dirs.run_wall_time_us":366,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:37.701294 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.012s user 0.008s sys 0.000s Metrics: {"bytes_written":272603,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":521,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20250113 03:10:37.701323 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1050740 bytes of WAL
I20250113 03:10:37.701440 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014606
I20250113 03:10:37.701558 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:37.701637 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000001 (ops 1-580)
I20250113 03:10:37.702078 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:37.702564 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014601
I20250113 03:10:37.728988 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.035s user 0.030s sys 0.001s Metrics: {"bytes_written":820694,"cfile_cache_hit":12,"cfile_cache_hit_bytes":556367,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1493417,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":485,"drs_written":1,"lbm_read_time_us":696,"lbm_reads_lt_1ms":74,"lbm_write_time_us":1298,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":435591,"rows_written":28900}
I20250113 03:10:37.729138 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 284012 bytes on disk
I20250113 03:10:37.729252 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.734874 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.032s user 0.031s sys 0.000s Metrics: {"bytes_written":839882,"cfile_cache_hit":12,"cfile_cache_hit_bytes":565267,"cfile_cache_miss":52,"cfile_cache_miss_bytes":1527479,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1206,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":402,"drs_written":1,"lbm_read_time_us":683,"lbm_reads_lt_1ms":76,"lbm_write_time_us":1392,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":444734,"rows_written":29500}
I20250113 03:10:37.734979 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 290072 bytes on disk
I20250113 03:10:37.735076 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.754441 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.049s user 0.030s sys 0.000s Metrics: {"bytes_written":824792,"cfile_cache_hit":12,"cfile_cache_hit_bytes":557942,"cfile_cache_miss":52,"cfile_cache_miss_bytes":1503068,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":300,"dirs.run_cpu_time_us":64,"dirs.run_wall_time_us":699,"drs_written":1,"lbm_read_time_us":768,"lbm_reads_lt_1ms":76,"lbm_write_time_us":1455,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":437869,"rows_written":29050}
I20250113 03:10:37.754581 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 285458 bytes on disk
I20250113 03:10:37.754707 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.840437 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.845793 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.846437 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.853757 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.004s sys 0.007s Metrics: {"bytes_written":283125,"cfile_init":1,"dirs.queue_time_us":114,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":620,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1355,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20250113 03:10:37.853910 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1049672 bytes of WAL
I20250113 03:10:37.853987 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:37.854025 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000001 (ops 1-621)
I20250113 03:10:37.854365 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:37.854403 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.009s user 0.008s sys 0.000s Metrics: {"bytes_written":286648,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":534,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20250113 03:10:37.854462 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 90998 bytes on disk
I20250113 03:10:37.854616 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1051756 bytes of WAL
I20250113 03:10:37.854627 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.854725 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014549
I20250113 03:10:37.854748 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:37.854777 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000001 (ops 1-621)
I20250113 03:10:37.855044 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:37.855111 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 92364 bytes on disk
I20250113 03:10:37.855207 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.855280 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014546
I20250113 03:10:37.860882 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":277811,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":412,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1682,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9650}
I20250113 03:10:37.861003 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89121 bytes on disk
I20250113 03:10:37.861095 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.861169 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014544
I20250113 03:10:37.913971 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.053s user 0.040s sys 0.001s Metrics: {"bytes_written":1109154,"cfile_cache_hit":12,"cfile_cache_hit_bytes":616487,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2177993,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":134,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":579,"drs_written":1,"lbm_read_time_us":1175,"lbm_reads_lt_1ms":82,"lbm_write_time_us":4715,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":591876,"rows_written":39150}
I20250113 03:10:37.914211 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 392075 bytes on disk
I20250113 03:10:37.914376 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.916610 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.062s user 0.041s sys 0.000s Metrics: {"bytes_written":1098752,"cfile_cache_hit":12,"cfile_cache_hit_bytes":616927,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2149683,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":216,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":922,"drs_written":1,"lbm_read_time_us":1238,"lbm_reads_lt_1ms":82,"lbm_write_time_us":2530,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":585776,"rows_written":38750}
I20250113 03:10:37.917891 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 387539 bytes on disk
I20250113 03:10:37.918329 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.925417 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.070s user 0.032s sys 0.005s Metrics: {"bytes_written":1106523,"cfile_cache_hit":13,"cfile_cache_hit_bytes":652265,"cfile_cache_miss":61,"cfile_cache_miss_bytes":2135209,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":14,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":614,"drs_written":1,"lbm_read_time_us":1477,"lbm_reads_lt_1ms":81,"lbm_write_time_us":2256,"lbm_writes_lt_1ms":65,"num_input_rowsets":2,"peak_mem_usage":590356,"rows_written":39050}
I20250113 03:10:37.925942 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 390946 bytes on disk
I20250113 03:10:37.926095 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:37.986811 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.989396 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:37.995409 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.006028 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.010s user 0.006s sys 0.003s Metrics: {"bytes_written":293488,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":474,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20250113 03:10:38.006191 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 94529 bytes on disk
I20250113 03:10:38.006325 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.006830 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014333
I20250113 03:10:38.007185 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.009s sys 0.002s Metrics: {"bytes_written":276389,"cfile_init":1,"dirs.queue_time_us":108,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":357,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20250113 03:10:38.007318 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 88468 bytes on disk
I20250113 03:10:38.007630 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.012s sys 0.000s Metrics: {"bytes_written":286729,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1671,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":10000}
I20250113 03:10:38.007810 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014339
I20250113 03:10:38.007889 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.008165 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014338
I20250113 03:10:38.062319 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.055s user 0.034s sys 0.015s Metrics: {"bytes_written":1395638,"cfile_cache_hit":10,"cfile_cache_hit_bytes":456122,"cfile_cache_miss":74,"cfile_cache_miss_bytes":3081896,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":750,"drs_written":1,"lbm_read_time_us":1347,"lbm_reads_lt_1ms":94,"lbm_write_time_us":3103,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":748166,"rows_written":49400}
I20250113 03:10:38.062490 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 500171 bytes on disk
I20250113 03:10:38.062598 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.064575 30889 maintenance_manager.cc:418] P 6b45c552a0d3422ba326065fa2203da9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.003577
I20250113 03:10:38.067039 30888 maintenance_manager.cc:642] P 6b45c552a0d3422ba326065fa2203da9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"bytes_written":6628,"cfile_init":1,"dirs.queue_time_us":102,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":523,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":148,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":5,"thread_start_us":60,"threads_started":1}
I20250113 03:10:38.067167 30889 maintenance_manager.cc:418] P 6b45c552a0d3422ba326065fa2203da9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 481 bytes on disk
I20250113 03:10:38.067374 30888 maintenance_manager.cc:642] P 6b45c552a0d3422ba326065fa2203da9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.075419 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.067s user 0.043s sys 0.004s Metrics: {"bytes_written":1376291,"cfile_cache_hit":12,"cfile_cache_hit_bytes":667807,"cfile_cache_miss":72,"cfile_cache_miss_bytes":2816371,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1391,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1152,"drs_written":1,"lbm_read_time_us":1990,"lbm_reads_lt_1ms":92,"lbm_write_time_us":2682,"lbm_writes_lt_1ms":76,"mutex_wait_us":630,"num_input_rowsets":2,"peak_mem_usage":736731,"rows_written":48650}
I20250113 03:10:38.075771 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 492487 bytes on disk
I20250113 03:10:38.075973 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.098636 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.091s user 0.042s sys 0.008s Metrics: {"bytes_written":1378580,"cfile_cache_hit":13,"cfile_cache_hit_bytes":704980,"cfile_cache_miss":69,"cfile_cache_miss_bytes":2784972,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":133,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":613,"drs_written":1,"lbm_read_time_us":1459,"lbm_reads_lt_1ms":93,"lbm_write_time_us":2321,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":738263,"rows_written":48750}
I20250113 03:10:38.098786 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 493483 bytes on disk
I20250113 03:10:38.098929 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.147089 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.149549 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.158687 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":279064,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":516,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:38.158878 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89283 bytes on disk
I20250113 03:10:38.159032 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.159328 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014161
I20250113 03:10:38.163697 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.175572 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.012s user 0.009s sys 0.000s Metrics: {"bytes_written":275132,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":614,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20250113 03:10:38.175678 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 87899 bytes on disk
I20250113 03:10:38.175772 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.175882 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014154
I20250113 03:10:38.178835 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.009s sys 0.000s Metrics: {"bytes_written":280331,"cfile_init":1,"dirs.queue_time_us":1209,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":950,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20250113 03:10:38.178937 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89784 bytes on disk
I20250113 03:10:38.179034 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.179150 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014159
I20250113 03:10:38.218742 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.059s user 0.043s sys 0.011s Metrics: {"bytes_written":1649077,"cfile_cache_hit":12,"cfile_cache_hit_bytes":721335,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3463019,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":593,"drs_written":1,"lbm_read_time_us":1447,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3130,"lbm_writes_lt_1ms":88,"num_input_rowsets":2,"peak_mem_usage":884638,"rows_written":58350,"spinlock_wait_cycles":384}
I20250113 03:10:38.218972 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 594671 bytes on disk
I20250113 03:10:38.219111 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.258420 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.082s user 0.052s sys 0.001s Metrics: {"bytes_written":1663947,"cfile_cache_hit":12,"cfile_cache_hit_bytes":722233,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3505551,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":424,"drs_written":1,"lbm_read_time_us":1374,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3029,"lbm_writes_lt_1ms":88,"num_input_rowsets":2,"peak_mem_usage":893788,"rows_written":58950}
I20250113 03:10:38.258723 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1049733 bytes of WAL
I20250113 03:10:38.259778 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.259848 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000002 (ops 581-1158)
I20250113 03:10:38.260696 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.260875 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 601677 bytes on disk
I20250113 03:10:38.260978 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.268783 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.090s user 0.051s sys 0.003s Metrics: {"bytes_written":1652423,"cfile_cache_hit":12,"cfile_cache_hit_bytes":722262,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3472198,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":134,"dirs.run_cpu_time_us":68,"dirs.run_wall_time_us":615,"drs_written":1,"lbm_read_time_us":3987,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":3864,"lbm_writes_lt_1ms":88,"num_input_rowsets":2,"peak_mem_usage":886923,"rows_written":58500}
I20250113 03:10:38.268939 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 596789 bytes on disk
I20250113 03:10:38.269052 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.330188 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.330296 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.332193 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.341285 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":293407,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":597,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20250113 03:10:38.341411 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1053578 bytes of WAL
I20250113 03:10:38.344318 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.344357 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000002 (ops 622-1251)
I20250113 03:10:38.345490 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.345883 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014121
I20250113 03:10:38.347971 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.007s sys 0.004s Metrics: {"bytes_written":288192,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":320,"drs_written":1,"lbm_read_time_us":531,"lbm_reads_lt_1ms":4,"lbm_write_time_us":540,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10050}
I20250113 03:10:38.348142 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1048857 bytes of WAL
I20250113 03:10:38.348255 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.348310 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000002 (ops 622-1236)
I20250113 03:10:38.348806 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.008s sys 0.003s Metrics: {"bytes_written":279146,"cfile_init":1,"dirs.queue_time_us":14,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":556,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1374,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:38.348971 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89407 bytes on disk
I20250113 03:10:38.349114 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.349251 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014119
I20250113 03:10:38.350136 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.350241 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 92727 bytes on disk
I20250113 03:10:38.350373 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.350508 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014121
I20250113 03:10:38.430824 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.080s user 0.057s sys 0.008s Metrics: {"bytes_written":1936910,"cfile_cache_hit":11,"cfile_cache_hit_bytes":483764,"cfile_cache_miss":95,"cfile_cache_miss_bytes":4437184,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":709,"drs_written":1,"lbm_read_time_us":1952,"lbm_reads_lt_1ms":115,"lbm_write_time_us":3301,"lbm_writes_lt_1ms":100,"num_input_rowsets":2,"peak_mem_usage":1040163,"rows_written":68550,"spinlock_wait_cycles":384}
I20250113 03:10:38.431051 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 703971 bytes on disk
I20250113 03:10:38.431612 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.473642 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.128s user 0.059s sys 0.005s Metrics: {"bytes_written":1938129,"cfile_cache_hit":11,"cfile_cache_hit_bytes":488127,"cfile_cache_miss":95,"cfile_cache_miss_bytes":4435967,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":122,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":978,"drs_written":1,"lbm_read_time_us":2204,"lbm_reads_lt_1ms":119,"lbm_write_time_us":3335,"lbm_writes_lt_1ms":100,"num_input_rowsets":2,"peak_mem_usage":1040928,"rows_written":68600}
I20250113 03:10:38.474130 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 704216 bytes on disk
I20250113 03:10:38.474243 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.484025 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.135s user 0.063s sys 0.000s Metrics: {"bytes_written":1939436,"cfile_cache_hit":13,"cfile_cache_hit_bytes":805514,"cfile_cache_miss":93,"cfile_cache_miss_bytes":4123526,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":533,"drs_written":1,"lbm_read_time_us":8144,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":111,"lbm_write_time_us":8742,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":98,"num_input_rowsets":2,"peak_mem_usage":1041693,"rows_written":68650}
I20250113 03:10:38.484246 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 704833 bytes on disk
I20250113 03:10:38.484562 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.502554 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.504593 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.505283 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.521013 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.009s sys 0.001s Metrics: {"bytes_written":293491,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":67,"dirs.run_wall_time_us":607,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20250113 03:10:38.521179 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014182
I20250113 03:10:38.522102 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.011s sys 0.000s Metrics: {"bytes_written":286823,"cfile_init":1,"dirs.queue_time_us":6502,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":511,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20250113 03:10:38.522282 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013911
I20250113 03:10:38.527066 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.008s sys 0.003s Metrics: {"bytes_written":293512,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":381,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1181,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20250113 03:10:38.527817 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.014183
I20250113 03:10:38.609081 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.081s user 0.072s sys 0.000s Metrics: {"bytes_written":2223612,"cfile_cache_hit":10,"cfile_cache_hit_bytes":456480,"cfile_cache_miss":108,"cfile_cache_miss_bytes":5215126,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":546,"drs_written":1,"lbm_read_time_us":2043,"lbm_reads_lt_1ms":132,"lbm_write_time_us":3562,"lbm_writes_lt_1ms":112,"num_input_rowsets":2,"peak_mem_usage":1197996,"rows_written":78850}
I20250113 03:10:38.609396 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 812924 bytes on disk
I20250113 03:10:38.609849 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.670661 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.674384 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.153s user 0.071s sys 0.001s Metrics: {"bytes_written":2225096,"cfile_cache_hit":11,"cfile_cache_hit_bytes":488324,"cfile_cache_miss":107,"cfile_cache_miss_bytes":5187080,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":213,"dirs.run_cpu_time_us":64,"dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":6285,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":130,"lbm_write_time_us":15082,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":108,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":1198811,"rows_written":78900,"spinlock_wait_cycles":7105792}
I20250113 03:10:38.674667 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.680270 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.158s user 0.063s sys 0.008s Metrics: {"bytes_written":2211903,"cfile_cache_hit":13,"cfile_cache_hit_bytes":845974,"cfile_cache_miss":105,"cfile_cache_miss_bytes":4804636,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":215,"dirs.run_cpu_time_us":629,"dirs.run_wall_time_us":7872,"drs_written":1,"lbm_read_time_us":4132,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":128,"lbm_write_time_us":3613,"lbm_writes_lt_1ms":111,"num_input_rowsets":2,"peak_mem_usage":1193376,"rows_written":78550}
I20250113 03:10:38.680518 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 810196 bytes on disk
I20250113 03:10:38.680706 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.680846 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.686723 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.011s sys 0.000s Metrics: {"bytes_written":275007,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":521,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20250113 03:10:38.687700 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89428 bytes on disk
I20250113 03:10:38.687884 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.688014 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013902
I20250113 03:10:38.693028 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.008s sys 0.003s Metrics: {"bytes_written":277846,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":63,"dirs.run_wall_time_us":501,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20250113 03:10:38.693181 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1051557 bytes of WAL
I20250113 03:10:38.693288 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.693354 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000003 (ops 1159-1737)
I20250113 03:10:38.694168 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.694288 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013901
I20250113 03:10:38.700353 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.004s sys 0.007s Metrics: {"bytes_written":292128,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":65,"dirs.run_wall_time_us":561,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10200}
I20250113 03:10:38.700500 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013933
I20250113 03:10:38.791607 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.097s user 0.076s sys 0.005s Metrics: {"bytes_written":2490551,"cfile_cache_hit":10,"cfile_cache_hit_bytes":444973,"cfile_cache_miss":120,"cfile_cache_miss_bytes":5930827,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":125,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":651,"drs_written":1,"lbm_read_time_us":2654,"lbm_reads_lt_1ms":148,"lbm_write_time_us":4977,"lbm_writes_lt_1ms":123,"num_input_rowsets":2,"peak_mem_usage":1356211,"rows_written":88550}
I20250113 03:10:38.791875 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 920430 bytes on disk
I20250113 03:10:38.792183 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.832755 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.835314 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.134s user 0.078s sys 0.003s Metrics: {"bytes_written":2495810,"cfile_cache_hit":11,"cfile_cache_hit_bytes":487311,"cfile_cache_miss":119,"cfile_cache_miss_bytes":5902491,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":127,"dirs.run_cpu_time_us":742,"dirs.run_wall_time_us":6120,"drs_written":1,"lbm_read_time_us":4715,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":142,"lbm_write_time_us":7379,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1358911,"rows_written":88750}
I20250113 03:10:38.837229 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.856650 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.012s sys 0.000s Metrics: {"bytes_written":283126,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":58,"dirs.run_wall_time_us":459,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20250113 03:10:38.856851 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1052178 bytes of WAL
I20250113 03:10:38.856961 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.857026 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000003 (ops 1237-1865)
I20250113 03:10:38.857406 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.857487 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1013689 bytes on disk
I20250113 03:10:38.857609 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":27,"lbm_reads_lt_1ms":8}
I20250113 03:10:38.857743 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013982
I20250113 03:10:38.859434 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.027s user 0.011s sys 0.000s Metrics: {"bytes_written":281665,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":60,"dirs.run_wall_time_us":642,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:38.859670 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013985
I20250113 03:10:38.905087 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.217s user 0.071s sys 0.011s Metrics: {"bytes_written":2486300,"cfile_cache_hit":12,"cfile_cache_hit_bytes":502707,"cfile_cache_miss":118,"cfile_cache_miss_bytes":5861033,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":627,"drs_written":1,"lbm_read_time_us":3361,"lbm_reads_lt_1ms":138,"lbm_write_time_us":7774,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1353871,"rows_written":88400}
I20250113 03:10:38.905344 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 918921 bytes on disk
I20250113 03:10:38.905507 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.905665 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:38.930910 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":386074,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":68,"dirs.run_wall_time_us":641,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":922,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13450}
I20250113 03:10:38.931100 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1055838 bytes of WAL
I20250113 03:10:38.931195 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:38.931267 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000003 (ops 1252-1887)
I20250113 03:10:38.931668 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:38.932420 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 125901 bytes on disk
I20250113 03:10:38.932598 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20250113 03:10:38.932731 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013949
I20250113 03:10:38.978138 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.120s user 0.072s sys 0.017s Metrics: {"bytes_written":2772649,"cfile_cache_hit":11,"cfile_cache_hit_bytes":479724,"cfile_cache_miss":131,"cfile_cache_miss_bytes":6626146,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":126,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":1074,"drs_written":1,"lbm_read_time_us":2747,"lbm_reads_lt_1ms":151,"lbm_write_time_us":5767,"lbm_writes_lt_1ms":135,"num_input_rowsets":2,"peak_mem_usage":1519311,"rows_written":98600}
I20250113 03:10:38.978300 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1035598 bytes on disk
I20250113 03:10:38.978441 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.029594 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.037649 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.178s user 0.087s sys 0.003s Metrics: {"bytes_written":2765917,"cfile_cache_hit":13,"cfile_cache_hit_bytes":604139,"cfile_cache_miss":129,"cfile_cache_miss_bytes":6482653,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":737,"drs_written":1,"lbm_read_time_us":2713,"lbm_reads_lt_1ms":153,"lbm_write_time_us":9837,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":132,"num_input_rowsets":2,"peak_mem_usage":1515301,"rows_written":98350}
I20250113 03:10:39.037879 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.052600 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.120s user 0.086s sys 0.008s Metrics: {"bytes_written":2862332,"cfile_cache_hit":12,"cfile_cache_hit_bytes":600351,"cfile_cache_miss":134,"cfile_cache_miss_bytes":6738181,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":245,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":2452,"lbm_reads_lt_1ms":154,"lbm_write_time_us":5227,"lbm_writes_lt_1ms":139,"num_input_rowsets":2,"peak_mem_usage":1568512,"rows_written":101850,"spinlock_wait_cycles":2233472}
I20250113 03:10:39.052807 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1072897 bytes on disk
I20250113 03:10:39.052932 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.062259 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.013s sys 0.000s Metrics: {"bytes_written":339535,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":534,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3219,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11850}
I20250113 03:10:39.062531 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1143356 bytes on disk
I20250113 03:10:39.062865 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":26,"lbm_reads_lt_1ms":8}
I20250113 03:10:39.063009 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013812
I20250113 03:10:39.063139 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.094834 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.064s user 0.013s sys 0.000s Metrics: {"bytes_written":300329,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":61,"dirs.run_wall_time_us":433,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1939,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10350}
I20250113 03:10:39.097191 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013824
I20250113 03:10:39.116169 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.051s user 0.012s sys 0.000s Metrics: {"bytes_written":293450,"cfile_init":1,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":617,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":28652,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":10250}
I20250113 03:10:39.116310 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 94502 bytes on disk
I20250113 03:10:39.116426 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.116545 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013443
I20250113 03:10:39.158887 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.096s user 0.085s sys 0.008s Metrics: {"bytes_written":3098163,"cfile_cache_hit":10,"cfile_cache_hit_bytes":487586,"cfile_cache_miss":146,"cfile_cache_miss_bytes":7472972,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":506,"drs_written":1,"lbm_read_time_us":2798,"lbm_reads_lt_1ms":166,"lbm_write_time_us":5072,"lbm_writes_lt_1ms":148,"num_input_rowsets":2,"peak_mem_usage":1705799,"rows_written":110200,"spinlock_wait_cycles":384}
I20250113 03:10:39.159256 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1167121 bytes on disk
I20250113 03:10:39.159409 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.220577 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.250499 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.030s user 0.011s sys 0.000s Metrics: {"bytes_written":273757,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":607,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20250113 03:10:39.250710 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 87593 bytes on disk
I20250113 03:10:39.250849 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.250981 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013354
I20250113 03:10:39.252645 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.136s user 0.089s sys 0.008s Metrics: {"bytes_written":3149826,"cfile_cache_hit":12,"cfile_cache_hit_bytes":652719,"cfile_cache_miss":146,"cfile_cache_miss_bytes":7448337,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":580,"drs_written":1,"lbm_read_time_us":5157,"lbm_reads_lt_1ms":166,"lbm_write_time_us":7016,"lbm_writes_lt_1ms":151,"num_input_rowsets":2,"peak_mem_usage":1738269,"rows_written":112100}
I20250113 03:10:39.252826 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1188800 bytes on disk
I20250113 03:10:39.252955 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.263183 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.296993 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.033s user 0.010s sys 0.001s Metrics: {"bytes_written":288191,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":532,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10050}
I20250113 03:10:39.297241 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013198
I20250113 03:10:39.316393 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.219s user 0.103s sys 0.000s Metrics: {"bytes_written":3061458,"cfile_cache_hit":12,"cfile_cache_hit_bytes":667692,"cfile_cache_miss":142,"cfile_cache_miss_bytes":7203142,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":437,"drs_written":1,"lbm_read_time_us":3778,"lbm_reads_lt_1ms":166,"lbm_write_time_us":7904,"lbm_writes_lt_1ms":146,"num_input_rowsets":2,"peak_mem_usage":1686986,"rows_written":108950}
I20250113 03:10:39.316515 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.338282 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":458863,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":482,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":875,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16000}
I20250113 03:10:39.339051 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1304030 bytes on disk
I20250113 03:10:39.339262 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20250113 03:10:39.339425 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013675
I20250113 03:10:39.409862 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.159s user 0.091s sys 0.020s Metrics: {"bytes_written":3359295,"cfile_cache_hit":13,"cfile_cache_hit_bytes":712227,"cfile_cache_miss":153,"cfile_cache_miss_bytes":7954567,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":422,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":676,"drs_written":1,"lbm_read_time_us":5226,"lbm_reads_lt_1ms":173,"lbm_write_time_us":9139,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":157,"num_input_rowsets":2,"peak_mem_usage":1862515,"rows_written":119700}
I20250113 03:10:39.410029 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1047940 bytes of WAL
I20250113 03:10:39.410101 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:39.410142 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000004 (ops 1738-2314)
I20250113 03:10:39.410486 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:39.410612 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1274345 bytes on disk
I20250113 03:10:39.410733 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.431114 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.454675 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.010s sys 0.002s Metrics: {"bytes_written":281671,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":62,"dirs.run_wall_time_us":750,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1042,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:39.455374 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013562
I20250113 03:10:39.463773 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.124s user 0.106s sys 0.006s Metrics: {"bytes_written":3510106,"cfile_cache_hit":11,"cfile_cache_hit_bytes":574241,"cfile_cache_miss":161,"cfile_cache_miss_bytes":8463257,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":400,"drs_written":1,"lbm_read_time_us":5473,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":180,"lbm_write_time_us":6719,"lbm_writes_lt_1ms":166,"num_input_rowsets":2,"peak_mem_usage":1941310,"rows_written":124950,"spinlock_wait_cycles":384}
I20250113 03:10:39.463971 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1332953 bytes on disk
I20250113 03:10:39.463984 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.166s user 0.097s sys 0.007s Metrics: {"bytes_written":3431087,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1053538,"cfile_cache_miss":156,"cfile_cache_miss_bytes":7790556,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":561,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":493,"drs_written":1,"lbm_read_time_us":7520,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":178,"lbm_write_time_us":6660,"lbm_writes_lt_1ms":162,"num_input_rowsets":2,"peak_mem_usage":1901769,"rows_written":122150,"spinlock_wait_cycles":384}
I20250113 03:10:39.464238 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1301569 bytes on disk
I20250113 03:10:39.464392 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.464109 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.494861 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.511155 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":281527,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":682,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:39.511278 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1048374 bytes of WAL
I20250113 03:10:39.511790 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:39.511862 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000004 (ops 1888-2522)
I20250113 03:10:39.512455 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:39.512581 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 90165 bytes on disk
I20250113 03:10:39.512696 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.512801 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013549
I20250113 03:10:39.546648 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.566433 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.011s sys 0.000s Metrics: {"bytes_written":280205,"cfile_init":1,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":566,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20250113 03:10:39.566633 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1063151 bytes of WAL
I20250113 03:10:39.566736 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:39.566814 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000004 (ops 1866-2504)
I20250113 03:10:39.567292 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:39.567340 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.111s user 0.095s sys 0.008s Metrics: {"bytes_written":3632395,"cfile_cache_hit":13,"cfile_cache_hit_bytes":912039,"cfile_cache_miss":165,"cfile_cache_miss_bytes":8471403,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":134,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":628,"drs_written":1,"lbm_read_time_us":3073,"lbm_reads_lt_1ms":189,"lbm_write_time_us":6012,"lbm_writes_lt_1ms":171,"num_input_rowsets":2,"peak_mem_usage":2021450,"rows_written":129500}
I20250113 03:10:39.567411 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89675 bytes on disk
I20250113 03:10:39.567636 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1383623 bytes on disk
I20250113 03:10:39.567788 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.569561 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.569660 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013520
I20250113 03:10:39.638927 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.654742 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.142s user 0.109s sys 0.008s Metrics: {"bytes_written":3703985,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478861,"cfile_cache_miss":169,"cfile_cache_miss_bytes":9085683,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":731,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1011,"drs_written":1,"lbm_read_time_us":3744,"lbm_reads_lt_1ms":189,"lbm_write_time_us":7070,"lbm_writes_lt_1ms":176,"mutex_wait_us":324,"num_input_rowsets":2,"peak_mem_usage":2061264,"rows_written":131950,"spinlock_wait_cycles":768}
I20250113 03:10:39.654906 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1411490 bytes on disk
I20250113 03:10:39.655064 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.655747 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.011s sys 0.000s Metrics: {"bytes_written":284347,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":616,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20250113 03:10:39.655952 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013486
I20250113 03:10:39.706279 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.705391 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.136s user 0.112s sys 0.000s Metrics: {"bytes_written":3778670,"cfile_cache_hit":13,"cfile_cache_hit_bytes":793308,"cfile_cache_miss":171,"cfile_cache_miss_bytes":8971964,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":491,"drs_written":1,"lbm_read_time_us":3350,"lbm_reads_lt_1ms":191,"lbm_write_time_us":6259,"lbm_writes_lt_1ms":178,"num_input_rowsets":2,"peak_mem_usage":2105993,"rows_written":134700,"spinlock_wait_cycles":1152}
I20250113 03:10:39.706414 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1441920 bytes on disk
I20250113 03:10:39.706517 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.719744 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.008s sys 0.001s Metrics: {"bytes_written":272531,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":589,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1002,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20250113 03:10:39.719910 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013453
I20250113 03:10:39.757418 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.786968 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.012s sys 0.000s Metrics: {"bytes_written":279137,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":67,"dirs.run_wall_time_us":752,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:39.787178 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 91082 bytes on disk
I20250113 03:10:39.787935 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.788326 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013423
I20250113 03:10:39.791085 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.135s user 0.116s sys 0.007s Metrics: {"bytes_written":3911288,"cfile_cache_hit":14,"cfile_cache_hit_bytes":998304,"cfile_cache_miss":176,"cfile_cache_miss_bytes":9107768,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":657,"drs_written":1,"lbm_read_time_us":4030,"lbm_reads_lt_1ms":200,"lbm_write_time_us":6784,"lbm_writes_lt_1ms":184,"num_input_rowsets":2,"peak_mem_usage":2182215,"rows_written":139400}
I20250113 03:10:39.791800 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1493562 bytes on disk
I20250113 03:10:39.792637 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.844122 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.856050 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.012s user 0.008s sys 0.000s Metrics: {"bytes_written":279183,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":501,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":409,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:39.856231 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1053432 bytes of WAL
I20250113 03:10:39.856601 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:39.856676 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000005 (ops 2315-2894)
I20250113 03:10:39.857064 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:39.857390 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89419 bytes on disk
I20250113 03:10:39.857645 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.857774 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013203
I20250113 03:10:39.867184 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.147s user 0.118s sys 0.005s Metrics: {"bytes_written":3964377,"cfile_cache_hit":12,"cfile_cache_hit_bytes":789470,"cfile_cache_miss":184,"cfile_cache_miss_bytes":9468676,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":587,"drs_written":1,"lbm_read_time_us":4108,"lbm_reads_lt_1ms":208,"lbm_write_time_us":7187,"lbm_writes_lt_1ms":186,"num_input_rowsets":2,"peak_mem_usage":2215159,"rows_written":141400}
I20250113 03:10:39.867378 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1517104 bytes on disk
I20250113 03:10:39.867534 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.898340 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:39.921905 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.011s sys 0.001s Metrics: {"bytes_written":280296,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":501,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":963,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20250113 03:10:39.922094 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89947 bytes on disk
I20250113 03:10:39.922454 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":217,"lbm_reads_lt_1ms":4}
I20250113 03:10:39.922608 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013171
I20250113 03:10:39.985430 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.197s user 0.120s sys 0.007s Metrics: {"bytes_written":4049492,"cfile_cache_hit":12,"cfile_cache_hit_bytes":465823,"cfile_cache_miss":186,"cfile_cache_miss_bytes":10013949,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1174,"drs_written":1,"lbm_read_time_us":5298,"lbm_reads_lt_1ms":206,"lbm_write_time_us":8179,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":189,"num_input_rowsets":2,"peak_mem_usage":2263663,"rows_written":144400,"spinlock_wait_cycles":2944}
I20250113 03:10:39.985697 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.002892 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":313677,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":817,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10850}
I20250113 03:10:40.003226 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013129
I20250113 03:10:40.058869 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.201s user 0.119s sys 0.012s Metrics: {"bytes_written":4182344,"cfile_cache_hit":15,"cfile_cache_hit_bytes":867399,"cfile_cache_miss":189,"cfile_cache_miss_bytes":9952935,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2583,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":490,"drs_written":1,"lbm_read_time_us":5017,"lbm_reads_lt_1ms":209,"lbm_write_time_us":19025,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":194,"mutex_wait_us":1296,"num_input_rowsets":2,"peak_mem_usage":2340031,"rows_written":149100,"spinlock_wait_cycles":768}
I20250113 03:10:40.059007 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1606596 bytes on disk
I20250113 03:10:40.059165 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.059245 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.080329 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.012s sys 0.000s Metrics: {"bytes_written":272607,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":552,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20250113 03:10:40.080557 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013090
I20250113 03:10:40.126137 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.203s user 0.114s sys 0.017s Metrics: {"bytes_written":4239437,"cfile_cache_hit":10,"cfile_cache_hit_bytes":447642,"cfile_cache_miss":196,"cfile_cache_miss_bytes":10521014,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":4454,"lbm_reads_lt_1ms":216,"lbm_write_time_us":7019,"lbm_writes_lt_1ms":197,"num_input_rowsets":2,"peak_mem_usage":2373290,"rows_written":151150}
I20250113 03:10:40.126315 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1630409 bytes on disk
I20250113 03:10:40.126456 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.136821 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.151248 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":272587,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":540,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":662,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20250113 03:10:40.151425 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1047069 bytes of WAL
I20250113 03:10:40.151528 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:40.151600 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000005 (ops 2523-3146)
I20250113 03:10:40.152019 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:40.152143 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 86926 bytes on disk
I20250113 03:10:40.152408 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.152575 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012991
I20250113 03:10:40.157613 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.154s user 0.131s sys 0.009s Metrics: {"bytes_written":4350158,"cfile_cache_hit":14,"cfile_cache_hit_bytes":745770,"cfile_cache_miss":196,"cfile_cache_miss_bytes":10526544,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":344,"dirs.run_cpu_time_us":79,"dirs.run_wall_time_us":681,"drs_written":1,"lbm_read_time_us":4749,"lbm_reads_lt_1ms":224,"lbm_write_time_us":8233,"lbm_writes_lt_1ms":202,"num_input_rowsets":2,"peak_mem_usage":2438808,"rows_written":155250,"spinlock_wait_cycles":384}
I20250113 03:10:40.158548 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1678006 bytes on disk
I20250113 03:10:40.159581 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.240739 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.257342 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.004s sys 0.007s Metrics: {"bytes_written":275013,"cfile_init":1,"dirs.queue_time_us":468,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":613,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20250113 03:10:40.257532 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1049082 bytes of WAL
I20250113 03:10:40.257715 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:40.257763 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000005 (ops 2505-3130)
I20250113 03:10:40.258245 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:40.258337 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013206
I20250113 03:10:40.265864 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.185s user 0.127s sys 0.016s Metrics: {"bytes_written":4442075,"cfile_cache_hit":11,"cfile_cache_hit_bytes":471590,"cfile_cache_miss":203,"cfile_cache_miss_bytes":11049238,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":770,"drs_written":1,"lbm_read_time_us":5049,"lbm_reads_lt_1ms":227,"lbm_write_time_us":8850,"lbm_writes_lt_1ms":206,"num_input_rowsets":2,"peak_mem_usage":2493831,"rows_written":158550,"spinlock_wait_cycles":768}
I20250113 03:10:40.265973 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1715707 bytes on disk
I20250113 03:10:40.266058 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.316632 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.329507 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.177s user 0.128s sys 0.012s Metrics: {"bytes_written":4499027,"cfile_cache_hit":10,"cfile_cache_hit_bytes":441985,"cfile_cache_miss":206,"cfile_cache_miss_bytes":11232511,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":456,"drs_written":1,"lbm_read_time_us":4114,"lbm_reads_lt_1ms":226,"lbm_write_time_us":7218,"lbm_writes_lt_1ms":208,"num_input_rowsets":2,"peak_mem_usage":2527140,"rows_written":160600,"spinlock_wait_cycles":768}
I20250113 03:10:40.329638 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1738863 bytes on disk
I20250113 03:10:40.329824 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.332432 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.010s sys 0.000s Metrics: {"bytes_written":275010,"cfile_init":1,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":637,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20250113 03:10:40.332675 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013200
I20250113 03:10:40.360236 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.377108 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.007s sys 0.004s Metrics: {"bytes_written":271070,"cfile_init":1,"dirs.queue_time_us":15,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":495,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":950,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20250113 03:10:40.377233 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 86474 bytes on disk
I20250113 03:10:40.377321 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.377390 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013194
I20250113 03:10:40.440439 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.182s user 0.127s sys 0.016s Metrics: {"bytes_written":4619052,"cfile_cache_hit":11,"cfile_cache_hit_bytes":473811,"cfile_cache_miss":211,"cfile_cache_miss_bytes":11507581,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":113,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":464,"drs_written":1,"lbm_read_time_us":4152,"lbm_reads_lt_1ms":235,"lbm_write_time_us":10397,"lbm_writes_lt_1ms":213,"num_input_rowsets":2,"peak_mem_usage":2595288,"rows_written":164800,"spinlock_wait_cycles":768}
I20250113 03:10:40.440603 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1787126 bytes on disk
I20250113 03:10:40.440693 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.492555 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.517452 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.182s user 0.126s sys 0.020s Metrics: {"bytes_written":4710370,"cfile_cache_hit":10,"cfile_cache_hit_bytes":444014,"cfile_cache_miss":216,"cfile_cache_miss_bytes":11780186,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":534,"drs_written":1,"lbm_read_time_us":5887,"lbm_reads_lt_1ms":240,"lbm_write_time_us":8494,"lbm_writes_lt_1ms":218,"num_input_rowsets":2,"peak_mem_usage":2648906,"rows_written":168100}
I20250113 03:10:40.518316 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1824684 bytes on disk
I20250113 03:10:40.518517 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.527154 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.034s user 0.009s sys 0.004s Metrics: {"bytes_written":281577,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":426,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1929,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9800}
I20250113 03:10:40.527732 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013145
I20250113 03:10:40.569444 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.192s user 0.143s sys 0.008s Metrics: {"bytes_written":4761210,"cfile_cache_hit":10,"cfile_cache_hit_bytes":441137,"cfile_cache_miss":218,"cfile_cache_miss_bytes":11924323,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":5352,"lbm_reads_lt_1ms":238,"lbm_write_time_us":10571,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":218,"num_input_rowsets":2,"peak_mem_usage":2679930,"rows_written":170000,"spinlock_wait_cycles":384}
I20250113 03:10:40.569613 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1846703 bytes on disk
I20250113 03:10:40.569733 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.570336 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.587117 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":277791,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":374,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20250113 03:10:40.587239 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 88920 bytes on disk
I20250113 03:10:40.587432 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.587775 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1049658 bytes of WAL
I20250113 03:10:40.587877 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:40.587906 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000006 (ops 2895-3472)
I20250113 03:10:40.588304 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:40.588388 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013105
I20250113 03:10:40.610539 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.643308 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.032s user 0.006s sys 0.006s Metrics: {"bytes_written":284380,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":471,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1837,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9900}
I20250113 03:10:40.643708 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012966
I20250113 03:10:40.732887 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.204s user 0.131s sys 0.016s Metrics: {"bytes_written":4892023,"cfile_cache_hit":13,"cfile_cache_hit_bytes":821688,"cfile_cache_miss":219,"cfile_cache_miss_bytes":11884384,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":691,"dirs.run_wall_time_us":4549,"drs_written":1,"lbm_read_time_us":6148,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":242,"lbm_write_time_us":10519,"lbm_writes_lt_1ms":225,"num_input_rowsets":2,"peak_mem_usage":2754278,"rows_written":174600,"spinlock_wait_cycles":768}
I20250113 03:10:40.733175 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 1899084 bytes on disk
I20250113 03:10:40.733430 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.733592 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.757853 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.014s sys 0.000s Metrics: {"bytes_written":351879,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":548,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":817,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12150}
I20250113 03:10:40.758039 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.013005
I20250113 03:10:40.799108 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.211s user 0.148s sys 0.012s Metrics: {"bytes_written":4976130,"cfile_cache_hit":12,"cfile_cache_hit_bytes":788055,"cfile_cache_miss":226,"cfile_cache_miss_bytes":12148855,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":352,"drs_written":1,"lbm_read_time_us":5462,"lbm_reads_lt_1ms":246,"lbm_write_time_us":8657,"lbm_writes_lt_1ms":227,"num_input_rowsets":2,"peak_mem_usage":2805606,"rows_written":177750,"spinlock_wait_cycles":768}
I20250113 03:10:40.799292 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.822759 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.013s sys 0.000s Metrics: {"bytes_written":354282,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":479,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12250}
I20250113 03:10:40.822966 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012985
I20250113 03:10:40.941293 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.183s user 0.155s sys 0.012s Metrics: {"bytes_written":5228021,"cfile_cache_hit":12,"cfile_cache_hit_bytes":835827,"cfile_cache_miss":234,"cfile_cache_miss_bytes":12761771,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":651,"drs_written":1,"lbm_read_time_us":5682,"lbm_reads_lt_1ms":258,"lbm_write_time_us":8488,"lbm_writes_lt_1ms":239,"num_input_rowsets":2,"peak_mem_usage":2949351,"rows_written":186750,"spinlock_wait_cycles":768}
I20250113 03:10:40.941586 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2037759 bytes on disk
I20250113 03:10:40.941747 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:40.982318 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:40.992792 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.170s user 0.139s sys 0.024s Metrics: {"bytes_written":5319347,"cfile_cache_hit":10,"cfile_cache_hit_bytes":496214,"cfile_cache_miss":238,"cfile_cache_miss_bytes":13341926,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":139,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":532,"drs_written":1,"lbm_read_time_us":5624,"lbm_reads_lt_1ms":266,"lbm_write_time_us":7789,"lbm_writes_lt_1ms":243,"num_input_rowsets":2,"peak_mem_usage":3002060,"rows_written":190000}
I20250113 03:10:40.993048 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2075258 bytes on disk
I20250113 03:10:40.993579 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.000375 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.008s sys 0.004s Metrics: {"bytes_written":272635,"cfile_init":1,"dirs.queue_time_us":27,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20250113 03:10:41.000521 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1061025 bytes of WAL
I20250113 03:10:41.000738 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.000769 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000006 (ops 3131-3755)
I20250113 03:10:41.001080 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:41.001158 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012930
I20250113 03:10:41.016142 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.372s user 0.143s sys 0.019s Metrics: {"bytes_written":5039794,"cfile_cache_hit":12,"cfile_cache_hit_bytes":792882,"cfile_cache_miss":228,"cfile_cache_miss_bytes":12299400,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":8902,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":250,"lbm_write_time_us":22947,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":228,"num_input_rowsets":2,"peak_mem_usage":2840295,"rows_written":179900}
I20250113 03:10:41.016273 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.045090 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.013s sys 0.006s Metrics: {"bytes_written":498242,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":596,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3250,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17400}
I20250113 03:10:41.045250 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1058022 bytes of WAL
I20250113 03:10:41.045392 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.045487 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000006 (ops 3147-3769)
I20250113 03:10:41.046114 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:41.046236 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012921
I20250113 03:10:41.065073 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.080400 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.011s sys 0.001s Metrics: {"bytes_written":279040,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":572,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:41.080550 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89451 bytes on disk
I20250113 03:10:41.080788 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.081089 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012839
I20250113 03:10:41.203145 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.202s user 0.154s sys 0.020s Metrics: {"bytes_written":5487681,"cfile_cache_hit":12,"cfile_cache_hit_bytes":730452,"cfile_cache_miss":246,"cfile_cache_miss_bytes":13563586,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1523,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":716,"drs_written":1,"lbm_read_time_us":6141,"lbm_reads_lt_1ms":270,"lbm_write_time_us":9139,"lbm_writes_lt_1ms":250,"num_input_rowsets":2,"peak_mem_usage":3105602,"rows_written":196200}
I20250113 03:10:41.204667 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2147277 bytes on disk
I20250113 03:10:41.205314 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.216521 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.234640 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.011s sys 0.000s Metrics: {"bytes_written":273745,"cfile_init":1,"dirs.queue_time_us":15,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":1083,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20250113 03:10:41.234948 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 87359 bytes on disk
I20250113 03:10:41.235098 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.235241 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012538
I20250113 03:10:41.326741 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.280s user 0.159s sys 0.018s Metrics: {"bytes_written":5519321,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1172887,"cfile_cache_miss":247,"cfile_cache_miss_bytes":13189027,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":860,"dirs.run_wall_time_us":8249,"drs_written":1,"lbm_read_time_us":8566,"lbm_reads_lt_1ms":275,"lbm_write_time_us":14426,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":252,"num_input_rowsets":2,"peak_mem_usage":3115517,"rows_written":197300,"spinlock_wait_cycles":768}
I20250113 03:10:41.326900 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2160233 bytes on disk
I20250113 03:10:41.327639 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.327776 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.339361 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.258s user 0.162s sys 0.015s Metrics: {"bytes_written":5585041,"cfile_cache_hit":11,"cfile_cache_hit_bytes":616405,"cfile_cache_miss":251,"cfile_cache_miss_bytes":13936103,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":1160,"drs_written":1,"lbm_read_time_us":8901,"lbm_reads_lt_1ms":271,"lbm_write_time_us":12267,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":254,"num_input_rowsets":2,"peak_mem_usage":3162225,"rows_written":199700,"spinlock_wait_cycles":384}
I20250113 03:10:41.339507 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2187296 bytes on disk
I20250113 03:10:41.339763 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.339867 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.355125 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":351894,"cfile_init":1,"dirs.queue_time_us":622,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2039,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":12150}
I20250113 03:10:41.355429 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1053488 bytes of WAL
I20250113 03:10:41.355573 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.355645 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000007 (ops 3473-4052)
I20250113 03:10:41.356312 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":14592}
I20250113 03:10:41.356472 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012047
I20250113 03:10:41.356622 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":407046,"cfile_init":1,"dirs.queue_time_us":18,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2151,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14100}
I20250113 03:10:41.356746 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 131885 bytes on disk
I20250113 03:10:41.357124 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.357229 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012546
I20250113 03:10:41.453186 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.218s user 0.156s sys 0.023s Metrics: {"bytes_written":5755359,"cfile_cache_hit":12,"cfile_cache_hit_bytes":640861,"cfile_cache_miss":258,"cfile_cache_miss_bytes":14352847,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":580,"drs_written":1,"lbm_read_time_us":8225,"lbm_reads_lt_1ms":278,"lbm_write_time_us":10067,"lbm_writes_lt_1ms":262,"num_input_rowsets":2,"peak_mem_usage":3259917,"rows_written":205700}
I20250113 03:10:41.453364 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.467995 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":299011,"cfile_init":1,"dirs.queue_time_us":13,"dirs.run_cpu_time_us":60,"dirs.run_wall_time_us":525,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10300}
I20250113 03:10:41.468463 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2352112 bytes on disk
I20250113 03:10:41.468652 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20250113 03:10:41.468768 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012163
I20250113 03:10:41.634027 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.277s user 0.166s sys 0.022s Metrics: {"bytes_written":5910184,"cfile_cache_hit":12,"cfile_cache_hit_bytes":864226,"cfile_cache_miss":268,"cfile_cache_miss_bytes":14541350,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":120,"dirs.run_cpu_time_us":558,"dirs.run_wall_time_us":3965,"drs_written":1,"lbm_read_time_us":9995,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":287,"lbm_write_time_us":12608,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":268,"num_input_rowsets":2,"peak_mem_usage":3347933,"rows_written":211400,"spinlock_wait_cycles":384}
I20250113 03:10:41.634182 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.655133 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.009s sys 0.004s Metrics: {"bytes_written":318878,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1670,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11050}
I20250113 03:10:41.655437 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1060892 bytes of WAL
I20250113 03:10:41.655591 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.655655 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000007 (ops 3770-4387)
I20250113 03:10:41.656361 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:41.656493 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012619
I20250113 03:10:41.662268 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.193s user 0.176s sys 0.012s Metrics: {"bytes_written":6040431,"cfile_cache_hit":12,"cfile_cache_hit_bytes":659667,"cfile_cache_miss":270,"cfile_cache_miss_bytes":15097187,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":579,"drs_written":1,"lbm_read_time_us":6505,"lbm_reads_lt_1ms":290,"lbm_write_time_us":9705,"lbm_writes_lt_1ms":274,"num_input_rowsets":2,"peak_mem_usage":3426483,"rows_written":216000,"spinlock_wait_cycles":2432}
I20250113 03:10:41.662576 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2374818 bytes on disk
I20250113 03:10:41.662773 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.673372 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.317s user 0.175s sys 0.019s Metrics: {"bytes_written":5921843,"cfile_cache_hit":12,"cfile_cache_hit_bytes":700720,"cfile_cache_miss":266,"cfile_cache_miss_bytes":14741916,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":548,"drs_written":1,"lbm_read_time_us":7853,"lbm_reads_lt_1ms":290,"lbm_write_time_us":18846,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":266,"num_input_rowsets":2,"peak_mem_usage":3357199,"rows_written":211850}
I20250113 03:10:41.673841 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.693967 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":342275,"cfile_init":1,"dirs.queue_time_us":284,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":1408,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":812,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11950}
I20250113 03:10:41.694108 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012606
I20250113 03:10:41.713780 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.737260 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.008s sys 0.004s Metrics: {"bytes_written":284420,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":535,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1394,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20250113 03:10:41.737418 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1064818 bytes of WAL
I20250113 03:10:41.737519 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.737571 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000007 (ops 3756-4386)
I20250113 03:10:41.739568 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:41.739679 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012582
I20250113 03:10:41.890419 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.234s user 0.177s sys 0.016s Metrics: {"bytes_written":6221115,"cfile_cache_hit":10,"cfile_cache_hit_bytes":473548,"cfile_cache_miss":282,"cfile_cache_miss_bytes":15754304,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":465,"dirs.run_wall_time_us":3736,"drs_written":1,"lbm_read_time_us":6634,"lbm_reads_lt_1ms":310,"lbm_write_time_us":13455,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":282,"num_input_rowsets":2,"peak_mem_usage":3530533,"rows_written":222450,"spinlock_wait_cycles":1536}
I20250113 03:10:41.890770 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2448861 bytes on disk
I20250113 03:10:41.890959 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:41.891096 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.906912 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.009s sys 0.004s Metrics: {"bytes_written":306835,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":481,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10600}
I20250113 03:10:41.907110 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012364
I20250113 03:10:41.944571 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.250s user 0.169s sys 0.024s Metrics: {"bytes_written":6254544,"cfile_cache_hit":13,"cfile_cache_hit_bytes":864067,"cfile_cache_miss":279,"cfile_cache_miss_bytes":15460523,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":636,"drs_written":1,"lbm_read_time_us":9560,"lbm_reads_lt_1ms":307,"lbm_write_time_us":15105,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":282,"num_input_rowsets":2,"peak_mem_usage":3551573,"rows_written":223800,"spinlock_wait_cycles":948224}
I20250113 03:10:41.944828 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:41.967628 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.013s sys 0.000s Metrics: {"bytes_written":319910,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":1557,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100}
I20250113 03:10:41.967928 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2568620 bytes on disk
I20250113 03:10:41.969359 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":86,"lbm_reads_lt_1ms":8}
I20250113 03:10:41.969517 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1053661 bytes of WAL
I20250113 03:10:41.969609 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:41.970050 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000008 (ops 4053-4632)
I20250113 03:10:41.970628 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:41.970746 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012347
I20250113 03:10:42.033098 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.293s user 0.180s sys 0.018s Metrics: {"bytes_written":6314888,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1320800,"cfile_cache_miss":278,"cfile_cache_miss_bytes":15163632,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":862,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":422,"drs_written":1,"lbm_read_time_us":8270,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":301,"lbm_write_time_us":15709,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":284,"mutex_wait_us":2,"num_input_rowsets":2,"peak_mem_usage":3587748,"rows_written":225900,"spinlock_wait_cycles":896}
I20250113 03:10:42.033401 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2487846 bytes on disk
I20250113 03:10:42.033576 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.033715 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.053038 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.011s sys 0.002s Metrics: {"bytes_written":339482,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":414,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11850}
I20250113 03:10:42.053246 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 110774 bytes on disk
I20250113 03:10:42.053400 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.053519 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012415
I20250113 03:10:42.121315 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.214s user 0.197s sys 0.008s Metrics: {"bytes_written":6515134,"cfile_cache_hit":11,"cfile_cache_hit_bytes":497837,"cfile_cache_miss":293,"cfile_cache_miss_bytes":16516327,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":459,"drs_written":1,"lbm_read_time_us":6430,"lbm_reads_lt_1ms":317,"lbm_write_time_us":9851,"lbm_writes_lt_1ms":294,"num_input_rowsets":2,"peak_mem_usage":3703224,"rows_written":233050}
I20250113 03:10:42.121584 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2571295 bytes on disk
I20250113 03:10:42.121729 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.162392 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.188443 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.026s user 0.011s sys 0.000s Metrics: {"bytes_written":276361,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":457,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20250113 03:10:42.188673 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012394
I20250113 03:10:42.219173 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.248s user 0.178s sys 0.024s Metrics: {"bytes_written":6565740,"cfile_cache_hit":15,"cfile_cache_hit_bytes":982767,"cfile_cache_miss":291,"cfile_cache_miss_bytes":16164885,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":793,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":588,"drs_written":1,"lbm_read_time_us":9681,"lbm_reads_lt_1ms":311,"lbm_write_time_us":12857,"lbm_writes_lt_1ms":296,"num_input_rowsets":2,"peak_mem_usage":3732782,"rows_written":234900,"spinlock_wait_cycles":1152}
I20250113 03:10:42.219401 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2592285 bytes on disk
I20250113 03:10:42.219523 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.221340 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.234411 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":279098,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":423,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1030,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:42.234565 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89174 bytes on disk
I20250113 03:10:42.234740 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.235105 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012374
I20250113 03:10:42.300122 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.247s user 0.191s sys 0.009s Metrics: {"bytes_written":6645326,"cfile_cache_hit":12,"cfile_cache_hit_bytes":692486,"cfile_cache_miss":296,"cfile_cache_miss_bytes":16658512,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":501,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":700,"drs_written":1,"lbm_read_time_us":8858,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":315,"lbm_write_time_us":9939,"lbm_writes_lt_1ms":300,"num_input_rowsets":2,"peak_mem_usage":3778341,"rows_written":237750,"spinlock_wait_cycles":1920}
I20250113 03:10:42.300645 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2625705 bytes on disk
I20250113 03:10:42.300807 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.302031 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.324514 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.013s sys 0.001s Metrics: {"bytes_written":276403,"cfile_init":1,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":535,"drs_written":1,"lbm_read_time_us":29,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20250113 03:10:42.324740 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 88273 bytes on disk
I20250113 03:10:42.324903 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.325094 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012180
I20250113 03:10:42.457886 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.223s user 0.209s sys 0.008s Metrics: {"bytes_written":6835633,"cfile_cache_hit":12,"cfile_cache_hit_bytes":781177,"cfile_cache_miss":304,"cfile_cache_miss_bytes":17083485,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":584,"drs_written":1,"lbm_read_time_us":7864,"lbm_reads_lt_1ms":324,"lbm_write_time_us":11931,"lbm_writes_lt_1ms":308,"num_input_rowsets":2,"peak_mem_usage":3891798,"rows_written":244600}
I20250113 03:10:42.458081 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2704239 bytes on disk
I20250113 03:10:42.458209 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.519073 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.330s user 0.190s sys 0.026s Metrics: {"bytes_written":6777509,"cfile_cache_hit":12,"cfile_cache_hit_bytes":642546,"cfile_cache_miss":302,"cfile_cache_miss_bytes":17080012,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":617,"drs_written":1,"lbm_read_time_us":9926,"lbm_reads_lt_1ms":326,"lbm_write_time_us":16277,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":304,"num_input_rowsets":2,"peak_mem_usage":3860214,"rows_written":242650,"spinlock_wait_cycles":768}
I20250113 03:10:42.519268 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.519716 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.535205 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.011s sys 0.003s Metrics: {"bytes_written":332871,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":1519,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11600}
I20250113 03:10:42.535351 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1138135 bytes of WAL
I20250113 03:10:42.535495 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:42.535532 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000008 (ops 4388-5047)
I20250113 03:10:42.535879 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:42.535962 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2789735 bytes on disk
I20250113 03:10:42.537204 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":1011,"lbm_reads_lt_1ms":8}
I20250113 03:10:42.537329 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012140
I20250113 03:10:42.542116 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.011s sys 0.000s Metrics: {"bytes_written":277751,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":867,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1817,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20250113 03:10:42.542796 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012137
I20250113 03:10:42.575428 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.250s user 0.187s sys 0.034s Metrics: {"bytes_written":6909677,"cfile_cache_hit":11,"cfile_cache_hit_bytes":612892,"cfile_cache_miss":309,"cfile_cache_miss_bytes":17454120,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":283,"dirs.run_cpu_time_us":79,"dirs.run_wall_time_us":12071,"drs_written":1,"lbm_read_time_us":8760,"lbm_reads_lt_1ms":329,"lbm_write_time_us":11980,"lbm_writes_lt_1ms":310,"num_input_rowsets":2,"peak_mem_usage":3936582,"rows_written":247350,"spinlock_wait_cycles":768}
I20250113 03:10:42.575718 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2736454 bytes on disk
I20250113 03:10:42.575911 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.606362 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.622979 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.011s sys 0.001s Metrics: {"bytes_written":279139,"cfile_init":1,"dirs.queue_time_us":10,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":494,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:42.623195 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1059205 bytes of WAL
I20250113 03:10:42.623423 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:42.623509 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000008 (ops 4387-5002)
I20250113 03:10:42.623919 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:42.624131 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89404 bytes on disk
I20250113 03:10:42.624490 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.624663 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011968
I20250113 03:10:42.754962 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.217s user 0.197s sys 0.012s Metrics: {"bytes_written":7099250,"cfile_cache_hit":12,"cfile_cache_hit_bytes":686518,"cfile_cache_miss":316,"cfile_cache_miss_bytes":17884610,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":6605,"lbm_reads_lt_1ms":336,"lbm_write_time_us":10754,"lbm_writes_lt_1ms":318,"num_input_rowsets":2,"peak_mem_usage":4046697,"rows_written":254250}
I20250113 03:10:42.755112 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2815716 bytes on disk
I20250113 03:10:42.755239 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.796702 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.807075 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.264s user 0.216s sys 0.003s Metrics: {"bytes_written":7099250,"cfile_cache_hit":12,"cfile_cache_hit_bytes":643424,"cfile_cache_miss":316,"cfile_cache_miss_bytes":17932368,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1601,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":777,"drs_written":1,"lbm_read_time_us":10590,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":339,"lbm_write_time_us":12550,"lbm_writes_lt_1ms":318,"num_input_rowsets":2,"peak_mem_usage":4048648,"rows_written":254250,"spinlock_wait_cycles":384}
I20250113 03:10:42.807260 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2815716 bytes on disk
I20250113 03:10:42.807422 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.807612 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.820873 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.008s sys 0.004s Metrics: {"bytes_written":279220,"cfile_init":1,"dirs.queue_time_us":14,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20250113 03:10:42.821236 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":299065,"cfile_init":1,"dirs.queue_time_us":19,"drs_written":1,"lbm_read_time_us":30,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10300}
I20250113 03:10:42.821429 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1093258 bytes of WAL
I20250113 03:10:42.821241 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011486
I20250113 03:10:42.821974 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:42.822060 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000009 (ops 4633-5234)
I20250113 03:10:42.822508 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:42.822901 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011480
I20250113 03:10:42.953576 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.328s user 0.192s sys 0.028s Metrics: {"bytes_written":7179877,"cfile_cache_hit":13,"cfile_cache_hit_bytes":809827,"cfile_cache_miss":317,"cfile_cache_miss_bytes":17973925,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1224,"dirs.run_cpu_time_us":901,"dirs.run_wall_time_us":9222,"drs_written":1,"lbm_read_time_us":10999,"lbm_reads_lt_1ms":337,"lbm_write_time_us":13946,"lbm_writes_lt_1ms":322,"num_input_rowsets":2,"peak_mem_usage":4094097,"rows_written":257050,"spinlock_wait_cycles":1433088}
I20250113 03:10:42.954231 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2847734 bytes on disk
I20250113 03:10:42.954641 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:42.956261 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:42.980036 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.006s sys 0.011s Metrics: {"bytes_written":374431,"cfile_init":1,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":533,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":13000}
I20250113 03:10:42.980258 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012146
I20250113 03:10:43.070895 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.248s user 0.226s sys 0.004s Metrics: {"bytes_written":7386262,"cfile_cache_hit":11,"cfile_cache_hit_bytes":625542,"cfile_cache_miss":327,"cfile_cache_miss_bytes":18706318,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1255,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":784,"drs_written":1,"lbm_read_time_us":8357,"lbm_reads_lt_1ms":351,"lbm_write_time_us":13012,"lbm_writes_lt_1ms":333,"num_input_rowsets":2,"peak_mem_usage":4215364,"rows_written":264550,"spinlock_wait_cycles":1536}
I20250113 03:10:43.071460 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2933787 bytes on disk
I20250113 03:10:43.071604 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.071768 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.089440 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":280281,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":481,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20250113 03:10:43.089589 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 89728 bytes on disk
I20250113 03:10:43.089718 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.089836 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012101
I20250113 03:10:43.244625 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.422s user 0.206s sys 0.024s Metrics: {"bytes_written":7371726,"cfile_cache_hit":14,"cfile_cache_hit_bytes":975918,"cfile_cache_miss":324,"cfile_cache_miss_bytes":18312624,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":1033,"drs_written":1,"lbm_read_time_us":15412,"lbm_reads_lt_1ms":348,"lbm_write_time_us":20176,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":330,"num_input_rowsets":2,"peak_mem_usage":4206214,"rows_written":263950,"spinlock_wait_cycles":768}
I20250113 03:10:43.244789 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2927068 bytes on disk
I20250113 03:10:43.244894 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.244971 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.277246 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.032s user 0.017s sys 0.001s Metrics: {"bytes_written":455350,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":592,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":837,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15850}
I20250113 03:10:43.277390 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.012045
I20250113 03:10:43.320073 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.339s user 0.225s sys 0.012s Metrics: {"bytes_written":7537725,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1023455,"cfile_cache_miss":330,"cfile_cache_miss_bytes":18715205,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":277,"dirs.run_cpu_time_us":965,"dirs.run_wall_time_us":8639,"drs_written":1,"lbm_read_time_us":9763,"lbm_reads_lt_1ms":354,"lbm_write_time_us":18071,"lbm_writes_lt_1ms":338,"num_input_rowsets":2,"peak_mem_usage":4302030,"rows_written":270050}
I20250113 03:10:43.322000 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 2997986 bytes on disk
I20250113 03:10:43.322444 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.322670 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.348222 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.025s user 0.013s sys 0.001s Metrics: {"bytes_written":351873,"cfile_init":1,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":503,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12150}
I20250113 03:10:43.348577 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1057085 bytes of WAL
I20250113 03:10:43.348775 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:43.348835 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000009 (ops 5003-5606)
I20250113 03:10:43.349633 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:43.349769 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011948
I20250113 03:10:43.377159 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.287s user 0.220s sys 0.021s Metrics: {"bytes_written":7658395,"cfile_cache_hit":11,"cfile_cache_hit_bytes":614354,"cfile_cache_miss":345,"cfile_cache_miss_bytes":19437766,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":450,"drs_written":1,"lbm_read_time_us":9501,"lbm_reads_lt_1ms":365,"lbm_write_time_us":15947,"lbm_writes_lt_1ms":343,"num_input_rowsets":2,"peak_mem_usage":4374350,"rows_written":274300,"spinlock_wait_cycles":1536}
I20250113 03:10:43.377341 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.391999 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":280382,"cfile_init":1,"dirs.queue_time_us":19,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":772,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20250113 03:10:43.392274 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3137275 bytes on disk
I20250113 03:10:43.392976 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20250113 03:10:43.393193 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011759
I20250113 03:10:43.609267 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.332s user 0.236s sys 0.012s Metrics: {"bytes_written":7809419,"cfile_cache_hit":11,"cfile_cache_hit_bytes":575966,"cfile_cache_miss":351,"cfile_cache_miss_bytes":19869574,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":632,"drs_written":1,"lbm_read_time_us":10541,"lbm_reads_lt_1ms":375,"lbm_write_time_us":16980,"lbm_writes_lt_1ms":350,"num_input_rowsets":2,"peak_mem_usage":4457603,"rows_written":279800,"spinlock_wait_cycles":1152}
I20250113 03:10:43.609493 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.631584 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":337407,"cfile_init":1,"dirs.queue_time_us":385,"dirs.run_cpu_time_us":55,"dirs.run_wall_time_us":337,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1993,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11750}
I20250113 03:10:43.632174 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1060084 bytes of WAL
I20250113 03:10:43.632256 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:43.632277 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000009 (ops 5048-5653)
I20250113 03:10:43.632615 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:43.632711 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3220014 bytes on disk
I20250113 03:10:43.632826 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":27,"lbm_reads_lt_1ms":8}
I20250113 03:10:43.632905 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011833
I20250113 03:10:43.690001 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.297s user 0.232s sys 0.028s Metrics: {"bytes_written":7929398,"cfile_cache_hit":12,"cfile_cache_hit_bytes":644597,"cfile_cache_miss":352,"cfile_cache_miss_bytes":20130081,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":682,"drs_written":1,"lbm_read_time_us":9512,"lbm_reads_lt_1ms":372,"lbm_write_time_us":15636,"lbm_writes_lt_1ms":355,"num_input_rowsets":2,"peak_mem_usage":4532764,"rows_written":284050,"spinlock_wait_cycles":1920}
I20250113 03:10:43.690342 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3160096 bytes on disk
I20250113 03:10:43.690656 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.690824 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.703876 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":289632,"cfile_init":1,"dirs.queue_time_us":39,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10100}
I20250113 03:10:43.704012 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1055386 bytes of WAL
I20250113 03:10:43.704073 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:43.704095 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000010 (ops 5235-5815)
I20250113 03:10:43.704463 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:43.704581 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011771
I20250113 03:10:43.896039 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.546s user 0.233s sys 0.028s Metrics: {"bytes_written":7877937,"cfile_cache_hit":12,"cfile_cache_hit_bytes":825847,"cfile_cache_miss":350,"cfile_cache_miss_bytes":19806031,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":532,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":767,"drs_written":1,"lbm_read_time_us":19849,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":373,"lbm_write_time_us":26737,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":352,"num_input_rowsets":2,"peak_mem_usage":4500304,"rows_written":282200,"thread_start_us":561,"threads_started":5}
I20250113 03:10:43.897399 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3138675 bytes on disk
I20250113 03:10:43.897670 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.897998 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.947501 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.314s user 0.223s sys 0.034s Metrics: {"bytes_written":8137416,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1018897,"cfile_cache_miss":360,"cfile_cache_miss_bytes":20304397,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":828,"drs_written":1,"lbm_read_time_us":9831,"lbm_reads_lt_1ms":380,"lbm_write_time_us":15913,"lbm_writes_lt_1ms":364,"mutex_wait_us":24,"num_input_rowsets":2,"peak_mem_usage":4652634,"rows_written":291550,"spinlock_wait_cycles":384}
I20250113 03:10:43.947762 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3246460 bytes on disk
I20250113 03:10:43.947926 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.948060 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:43.964560 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.012s sys 0.001s Metrics: {"bytes_written":292087,"cfile_init":1,"dirs.queue_time_us":743,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10200}
I20250113 03:10:43.964773 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011726
I20250113 03:10:43.973664 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.074s user 0.019s sys 0.001s Metrics: {"bytes_written":513541,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":57,"dirs.run_wall_time_us":8100,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3138,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":18000}
I20250113 03:10:43.973901 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 171172 bytes on disk
I20250113 03:10:43.974364 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:43.974548 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011769
I20250113 03:10:43.992875 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.288s user 0.244s sys 0.026s Metrics: {"bytes_written":8208750,"cfile_cache_hit":13,"cfile_cache_hit_bytes":817525,"cfile_cache_miss":363,"cfile_cache_miss_bytes":20700217,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":899,"drs_written":1,"lbm_read_time_us":9656,"lbm_reads_lt_1ms":387,"lbm_write_time_us":15498,"lbm_writes_lt_1ms":367,"num_input_rowsets":2,"peak_mem_usage":4696534,"rows_written":294150,"spinlock_wait_cycles":768}
I20250113 03:10:43.993898 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3276050 bytes on disk
I20250113 03:10:43.994052 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20250113 03:10:44.004887 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.024061 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.013s sys 0.000s Metrics: {"bytes_written":283238,"cfile_init":1,"dirs.queue_time_us":13,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":760,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1095,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20250113 03:10:44.024268 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 92140 bytes on disk
I20250113 03:10:44.025027 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20250113 03:10:44.025197 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011767
I20250113 03:10:44.305810 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.341s user 0.242s sys 0.029s Metrics: {"bytes_written":8419576,"cfile_cache_hit":13,"cfile_cache_hit_bytes":951697,"cfile_cache_miss":373,"cfile_cache_miss_bytes":21130765,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":95,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":746,"drs_written":1,"lbm_read_time_us":14642,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":396,"lbm_write_time_us":15567,"lbm_writes_lt_1ms":375,"num_input_rowsets":2,"peak_mem_usage":4819925,"rows_written":301750,"spinlock_wait_cycles":384}
I20250113 03:10:44.306134 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3363273 bytes on disk
I20250113 03:10:44.306318 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:44.306478 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.326659 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.013s sys 0.004s Metrics: {"bytes_written":379552,"cfile_init":1,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":527,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1427,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13200}
I20250113 03:10:44.326910 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1051977 bytes of WAL
I20250113 03:10:44.327025 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:44.327093 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000010 (ops 5654-6252)
I20250113 03:10:44.327665 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:44.327814 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011664
I20250113 03:10:44.357761 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.332s user 0.256s sys 0.028s Metrics: {"bytes_written":8481840,"cfile_cache_hit":13,"cfile_cache_hit_bytes":944071,"cfile_cache_miss":375,"cfile_cache_miss_bytes":21300009,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":1229,"drs_written":1,"lbm_read_time_us":10620,"lbm_reads_lt_1ms":395,"lbm_write_time_us":18420,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":376,"num_input_rowsets":2,"peak_mem_usage":4856829,"rows_written":304000,"spinlock_wait_cycles":384}
I20250113 03:10:44.358042 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.374593 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":360025,"cfile_init":1,"dirs.queue_time_us":22,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1001,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12450}
I20250113 03:10:44.374917 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011575
I20250113 03:10:44.421820 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.447s user 0.258s sys 0.021s Metrics: {"bytes_written":8376238,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1241616,"cfile_cache_miss":369,"cfile_cache_miss_bytes":20701210,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":321,"dirs.run_wall_time_us":14612,"drs_written":1,"lbm_read_time_us":21779,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":386,"lbm_write_time_us":20732,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":372,"num_input_rowsets":2,"peak_mem_usage":4786927,"rows_written":300200,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":3}
I20250113 03:10:44.422165 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3346196 bytes on disk
I20250113 03:10:44.426149 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:44.426326 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.458894 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.016s sys 0.004s Metrics: {"bytes_written":518567,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18200}
I20250113 03:10:44.459180 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1049452 bytes of WAL
I20250113 03:10:44.459319 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:44.459416 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000010 (ops 5607-6203)
I20250113 03:10:44.459852 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:44.459977 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011631
I20250113 03:10:44.673105 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.298s user 0.263s sys 0.026s Metrics: {"bytes_written":8829415,"cfile_cache_hit":12,"cfile_cache_hit_bytes":704019,"cfile_cache_miss":388,"cfile_cache_miss_bytes":22455507,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":223,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":657,"drs_written":1,"lbm_read_time_us":10176,"lbm_reads_lt_1ms":416,"lbm_write_time_us":16539,"lbm_writes_lt_1ms":392,"num_input_rowsets":2,"peak_mem_usage":5056532,"rows_written":316450,"spinlock_wait_cycles":785792}
I20250113 03:10:44.673625 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.688369 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.010s sys 0.002s Metrics: {"bytes_written":312186,"cfile_init":1,"dirs.queue_time_us":14,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10800}
I20250113 03:10:44.688597 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1207831 bytes of WAL
I20250113 03:10:44.689090 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:44.689247 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000011 (ops 5816-6480)
I20250113 03:10:44.689898 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:44.690038 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3633134 bytes on disk
I20250113 03:10:44.690236 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20250113 03:10:44.690372 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011460
I20250113 03:10:44.763185 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.435s user 0.269s sys 0.020s Metrics: {"bytes_written":8786233,"cfile_cache_hit":12,"cfile_cache_hit_bytes":720403,"cfile_cache_miss":388,"cfile_cache_miss_bytes":22324671,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":232,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":2634,"drs_written":1,"lbm_read_time_us":16093,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":410,"lbm_write_time_us":25010,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":388,"num_input_rowsets":2,"peak_mem_usage":5031401,"rows_written":314950,"spinlock_wait_cycles":1130880}
I20250113 03:10:44.763342 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.785579 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.018s sys 0.000s Metrics: {"bytes_written":431030,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1293,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":15050}
I20250113 03:10:44.785857 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011484
I20250113 03:10:44.944795 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.485s user 0.261s sys 0.025s Metrics: {"bytes_written":8883299,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1075715,"cfile_cache_miss":390,"cfile_cache_miss_bytes":22208077,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":875,"drs_written":1,"lbm_read_time_us":11782,"lbm_reads_lt_1ms":414,"lbm_write_time_us":18749,"lbm_writes_lt_1ms":395,"num_input_rowsets":2,"peak_mem_usage":5082458,"rows_written":318400,"spinlock_wait_cycles":5604608,"thread_start_us":176,"threads_started":5}
I20250113 03:10:44.945019 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:44.980450 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.035s user 0.021s sys 0.000s Metrics: {"bytes_written":518519,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":546,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2809,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":18200}
I20250113 03:10:44.980753 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3729209 bytes on disk
I20250113 03:10:44.981078 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":31,"lbm_reads_lt_1ms":8}
I20250113 03:10:44.981644 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011410
I20250113 03:10:45.001824 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.311s user 0.267s sys 0.020s Metrics: {"bytes_written":9126591,"cfile_cache_hit":12,"cfile_cache_hit_bytes":667214,"cfile_cache_miss":400,"cfile_cache_miss_bytes":23290648,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":762,"drs_written":1,"lbm_read_time_us":9546,"lbm_reads_lt_1ms":420,"lbm_write_time_us":16469,"lbm_writes_lt_1ms":405,"num_input_rowsets":2,"peak_mem_usage":5233673,"rows_written":327250,"spinlock_wait_cycles":3072}
I20250113 03:10:45.002163 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3658024 bytes on disk
I20250113 03:10:45.002314 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.002640 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.017880 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":332979,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11600}
I20250113 03:10:45.018087 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011430
I20250113 03:10:45.187577 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.402s user 0.266s sys 0.033s Metrics: {"bytes_written":9200420,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1227850,"cfile_cache_miss":401,"cfile_cache_miss_bytes":22921440,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":513,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1358,"drs_written":1,"lbm_read_time_us":12465,"lbm_reads_lt_1ms":429,"lbm_write_time_us":21142,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":408,"num_input_rowsets":2,"peak_mem_usage":5274100,"rows_written":330000,"spinlock_wait_cycles":768}
I20250113 03:10:45.188467 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3689875 bytes on disk
I20250113 03:10:45.189051 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.189435 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.218474 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.029s user 0.012s sys 0.008s Metrics: {"bytes_written":402135,"cfile_init":1,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":592,"drs_written":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1717,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":13900}
I20250113 03:10:45.218689 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1056451 bytes of WAL
I20250113 03:10:45.218823 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:45.218902 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000011 (ops 6253-6867)
I20250113 03:10:45.219450 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:45.219586 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 132234 bytes on disk
I20250113 03:10:45.219715 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.219851 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011285
I20250113 03:10:45.335201 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.317s user 0.279s sys 0.023s Metrics: {"bytes_written":9447566,"cfile_cache_hit":13,"cfile_cache_hit_bytes":975243,"cfile_cache_miss":415,"cfile_cache_miss_bytes":23837975,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":721,"drs_written":1,"lbm_read_time_us":9148,"lbm_reads_lt_1ms":439,"lbm_write_time_us":16050,"lbm_writes_lt_1ms":420,"num_input_rowsets":2,"peak_mem_usage":5421350,"rows_written":338850,"spinlock_wait_cycles":33920}
I20250113 03:10:45.335752 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3791660 bytes on disk
I20250113 03:10:45.335881 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.356439 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.373490 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.011s sys 0.000s Metrics: {"bytes_written":276396,"cfile_init":1,"dirs.queue_time_us":243,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":643,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1877,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20250113 03:10:45.373662 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 88170 bytes on disk
I20250113 03:10:45.373792 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.373888 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011314
I20250113 03:10:45.394445 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.413s user 0.259s sys 0.042s Metrics: {"bytes_written":9382265,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1236681,"cfile_cache_miss":411,"cfile_cache_miss_bytes":23392539,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":726,"drs_written":1,"lbm_read_time_us":12129,"lbm_reads_lt_1ms":431,"lbm_write_time_us":17553,"lbm_writes_lt_1ms":417,"num_input_rowsets":2,"peak_mem_usage":5378189,"rows_written":336600,"spinlock_wait_cycles":1920}
I20250113 03:10:45.394742 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3765501 bytes on disk
I20250113 03:10:45.395246 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.395401 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.415813 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.018s user 0.013s sys 0.002s Metrics: {"bytes_written":377073,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13100}
I20250113 03:10:45.416000 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1058343 bytes of WAL
I20250113 03:10:45.416096 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:45.416152 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000011 (ops 6204-6822)
I20250113 03:10:45.416592 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:45.416704 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011315
I20250113 03:10:45.624861 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.405s user 0.292s sys 0.019s Metrics: {"bytes_written":9589063,"cfile_cache_hit":13,"cfile_cache_hit_bytes":897110,"cfile_cache_miss":425,"cfile_cache_miss_bytes":24286444,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":785,"dirs.run_cpu_time_us":684,"dirs.run_wall_time_us":4208,"drs_written":1,"lbm_read_time_us":15144,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":444,"lbm_write_time_us":21121,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":425,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":5501107,"rows_written":343900,"spinlock_wait_cycles":196480}
I20250113 03:10:45.625633 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3850252 bytes on disk
I20250113 03:10:45.626034 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.626251 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.645112 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.019s user 0.010s sys 0.003s Metrics: {"bytes_written":327861,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":532,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1834,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11400}
I20250113 03:10:45.645339 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 105665 bytes on disk
I20250113 03:10:45.645484 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20250113 03:10:45.645617 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011252
I20250113 03:10:45.777011 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.360s user 0.281s sys 0.025s Metrics: {"bytes_written":9745460,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1043224,"cfile_cache_miss":430,"cfile_cache_miss_bytes":24559678,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":520,"drs_written":1,"lbm_read_time_us":10594,"lbm_reads_lt_1ms":454,"lbm_write_time_us":18340,"lbm_writes_lt_1ms":433,"num_input_rowsets":2,"peak_mem_usage":5596154,"rows_written":349700,"spinlock_wait_cycles":384}
I20250113 03:10:45.777920 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.794898 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":284282,"cfile_init":1,"dirs.queue_time_us":21,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1020,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20250113 03:10:45.795363 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011128
I20250113 03:10:45.806716 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.433s user 0.277s sys 0.030s Metrics: {"bytes_written":9710961,"cfile_cache_hit":14,"cfile_cache_hit_bytes":965792,"cfile_cache_miss":428,"cfile_cache_miss_bytes":24551280,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":79,"dirs.run_wall_time_us":669,"drs_written":1,"lbm_read_time_us":14963,"lbm_reads_lt_1ms":448,"lbm_write_time_us":20883,"lbm_writes_lt_1ms":431,"num_input_rowsets":2,"peak_mem_usage":5579347,"rows_written":348450,"spinlock_wait_cycles":1536}
I20250113 03:10:45.807099 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:45.827031 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":339601,"cfile_init":1,"dirs.queue_time_us":17,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1697,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11850}
I20250113 03:10:45.827206 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.011121
I20250113 03:10:46.010764 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.365s user 0.288s sys 0.027s Metrics: {"bytes_written":9900913,"cfile_cache_hit":12,"cfile_cache_hit_bytes":679917,"cfile_cache_miss":438,"cfile_cache_miss_bytes":25342301,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":663,"drs_written":1,"lbm_read_time_us":13241,"lbm_reads_lt_1ms":458,"lbm_write_time_us":17972,"lbm_writes_lt_1ms":438,"num_input_rowsets":2,"peak_mem_usage":5688847,"rows_written":355300,"spinlock_wait_cycles":20864}
I20250113 03:10:46.011125 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 3982168 bytes on disk
I20250113 03:10:46.011319 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:46.011456 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.028841 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":317406,"cfile_init":1,"dirs.queue_time_us":84,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":900,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11000}
I20250113 03:10:46.029063 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010933
I20250113 03:10:46.145643 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.350s user 0.257s sys 0.054s Metrics: {"bytes_written":10023073,"cfile_cache_hit":13,"cfile_cache_hit_bytes":808936,"cfile_cache_miss":443,"cfile_cache_miss_bytes":25528008,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":564,"dirs.run_wall_time_us":4344,"drs_written":1,"lbm_read_time_us":10171,"lbm_reads_lt_1ms":471,"lbm_write_time_us":18015,"lbm_writes_lt_1ms":444,"num_input_rowsets":2,"peak_mem_usage":5760221,"rows_written":359600,"spinlock_wait_cycles":1536}
I20250113 03:10:46.146449 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.163407 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":299067,"cfile_init":1,"dirs.queue_time_us":20,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1134,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10300}
I20250113 03:10:46.163633 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010968
I20250113 03:10:46.401233 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.372s user 0.287s sys 0.040s Metrics: {"bytes_written":10205706,"cfile_cache_hit":12,"cfile_cache_hit_bytes":670395,"cfile_cache_miss":450,"cfile_cache_miss_bytes":26163187,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":857,"drs_written":1,"lbm_read_time_us":12941,"lbm_reads_lt_1ms":474,"lbm_write_time_us":20585,"lbm_writes_lt_1ms":452,"num_input_rowsets":2,"peak_mem_usage":5867982,"rows_written":366300}
I20250113 03:10:46.401539 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.409065 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.582s user 0.289s sys 0.032s Metrics: {"bytes_written":10040648,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1176532,"cfile_cache_miss":441,"cfile_cache_miss_bytes":25210332,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":17,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":837,"drs_written":1,"lbm_read_time_us":18519,"lbm_reads_lt_1ms":469,"lbm_write_time_us":22989,"lbm_writes_lt_1ms":445,"num_input_rowsets":2,"peak_mem_usage":5769645,"rows_written":360300}
I20250113 03:10:46.409643 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1057034 bytes of WAL
I20250113 03:10:46.409816 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:46.409904 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000012 (ops 6481-7062)
I20250113 03:10:46.410420 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:46.410573 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.421885 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.020s user 0.015s sys 0.000s Metrics: {"bytes_written":331803,"cfile_init":1,"dirs.queue_time_us":12,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11550}
I20250113 03:10:46.422044 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1076469 bytes of WAL
I20250113 03:10:46.422173 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:46.422235 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000012 (ops 6868-7462)
I20250113 03:10:46.422670 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:46.422789 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4216879 bytes on disk
I20250113 03:10:46.422963 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20250113 03:10:46.423184 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010985
I20250113 03:10:46.445425 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.035s user 0.019s sys 0.000s Metrics: {"bytes_written":505861,"cfile_init":1,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1078,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17700}
I20250113 03:10:46.446355 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010985
I20250113 03:10:46.567235 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.403s user 0.290s sys 0.028s Metrics: {"bytes_written":10306963,"cfile_cache_hit":12,"cfile_cache_hit_bytes":655084,"cfile_cache_miss":454,"cfile_cache_miss_bytes":26448896,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":1226,"dirs.run_wall_time_us":9806,"drs_written":1,"lbm_read_time_us":13812,"lbm_reads_lt_1ms":482,"lbm_write_time_us":20145,"lbm_writes_lt_1ms":456,"num_input_rowsets":2,"peak_mem_usage":5927186,"rows_written":369900,"spinlock_wait_cycles":1920}
I20250113 03:10:46.567489 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4151209 bytes on disk
I20250113 03:10:46.568009 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:46.568200 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.592237 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.007s sys 0.008s Metrics: {"bytes_written":371761,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":445,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12900}
I20250113 03:10:46.592468 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1055514 bytes of WAL
I20250113 03:10:46.592607 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:46.592693 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000012 (ops 6823-7406)
I20250113 03:10:46.593626 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:46.593771 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010886
I20250113 03:10:46.799034 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.376s user 0.296s sys 0.042s Metrics: {"bytes_written":10522354,"cfile_cache_hit":11,"cfile_cache_hit_bytes":646479,"cfile_cache_miss":465,"cfile_cache_miss_bytes":27031779,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":725,"drs_written":1,"lbm_read_time_us":14605,"lbm_reads_lt_1ms":485,"lbm_write_time_us":19701,"lbm_writes_lt_1ms":465,"num_input_rowsets":2,"peak_mem_usage":6055111,"rows_written":377850,"spinlock_wait_cycles":768}
I20250113 03:10:46.799346 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:46.821480 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.014s sys 0.001s Metrics: {"bytes_written":351917,"cfile_init":1,"dirs.queue_time_us":188,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12150}
I20250113 03:10:46.821727 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4356074 bytes on disk
I20250113 03:10:46.822026 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":97,"lbm_reads_lt_1ms":8}
I20250113 03:10:46.822201 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010511
I20250113 03:10:46.973037 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.526s user 0.306s sys 0.025s Metrics: {"bytes_written":10530153,"cfile_cache_hit":13,"cfile_cache_hit_bytes":906320,"cfile_cache_miss":465,"cfile_cache_miss_bytes":26773528,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":746,"drs_written":1,"lbm_read_time_us":14153,"lbm_reads_lt_1ms":493,"lbm_write_time_us":25645,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":465,"num_input_rowsets":2,"peak_mem_usage":6051393,"rows_written":378000}
I20250113 03:10:46.973888 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4245102 bytes on disk
I20250113 03:10:46.974098 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:46.974224 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.000708 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.026s user 0.003s sys 0.016s Metrics: {"bytes_written":502050,"cfile_init":1,"dirs.queue_time_us":18,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1108,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17550}
I20250113 03:10:47.001387 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010566
I20250113 03:10:47.102480 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.509s user 0.314s sys 0.021s Metrics: {"bytes_written":10660624,"cfile_cache_hit":11,"cfile_cache_hit_bytes":550645,"cfile_cache_miss":469,"cfile_cache_miss_bytes":27493719,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":4119,"drs_written":1,"lbm_read_time_us":16185,"lbm_reads_lt_1ms":493,"lbm_write_time_us":25347,"lbm_writes_lt_1ms":470,"num_input_rowsets":2,"peak_mem_usage":6134189,"rows_written":382800,"spinlock_wait_cycles":384,"thread_start_us":673,"threads_started":5}
I20250113 03:10:47.102830 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4300301 bytes on disk
I20250113 03:10:47.103232 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.104298 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.128002 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.023s user 0.018s sys 0.000s Metrics: {"bytes_written":475406,"cfile_init":1,"dirs.queue_time_us":19,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16650}
I20250113 03:10:47.128227 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1056776 bytes of WAL
I20250113 03:10:47.128397 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:47.128475 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000013 (ops 7407-7988)
I20250113 03:10:47.128937 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:47.129065 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010524
I20250113 03:10:47.265166 20553 test_util.cc:274] Using random seed: -898349172
I20250113 03:10:47.277479 20553 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250113 03:10:47.290340 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.468s user 0.323s sys 0.027s Metrics: {"bytes_written":10861606,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1180319,"cfile_cache_miss":475,"cfile_cache_miss_bytes":27392155,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":738,"dirs.run_cpu_time_us":1708,"dirs.run_wall_time_us":14666,"drs_written":1,"lbm_read_time_us":15863,"lbm_reads_lt_1ms":495,"lbm_write_time_us":24470,"lbm_writes_lt_1ms":480,"num_input_rowsets":2,"peak_mem_usage":6251935,"rows_written":390000,"spinlock_wait_cycles":384}
I20250113 03:10:47.290738 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.323058 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.032s user 0.017s sys 0.000s Metrics: {"bytes_written":410746,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14250}
I20250113 03:10:47.323433 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1058556 bytes of WAL
I20250113 03:10:47.323519 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:47.323607 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000013 (ops 7463-8045)
I20250113 03:10:47.324432 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:47.324589 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4519057 bytes on disk
I20250113 03:10:47.325091 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":28,"lbm_reads_lt_1ms":8}
I20250113 03:10:47.325256 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010559
I20250113 03:10:47.455647 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.454s user 0.320s sys 0.034s Metrics: {"bytes_written":11014031,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1550471,"cfile_cache_miss":481,"cfile_cache_miss_bytes":27414441,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":1563,"drs_written":1,"lbm_read_time_us":16400,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":504,"lbm_write_time_us":22801,"lbm_writes_lt_1ms":487,"num_input_rowsets":2,"peak_mem_usage":6336704,"rows_written":395550,"spinlock_wait_cycles":2048}
I20250113 03:10:47.455878 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1084182 bytes of WAL
I20250113 03:10:47.455991 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:47.456058 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000013 (ops 7063-7659)
I20250113 03:10:47.456454 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:47.456600 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.487984 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.031s user 0.011s sys 0.007s Metrics: {"bytes_written":445365,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":586,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15450}
I20250113 03:10:47.488270 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010508
I20250113 03:10:47.639115 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.510s user 0.327s sys 0.040s Metrics: {"bytes_written":11122608,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1284036,"cfile_cache_miss":485,"cfile_cache_miss_bytes":27975944,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":597,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":708,"drs_written":1,"lbm_read_time_us":13147,"lbm_reads_lt_1ms":509,"lbm_write_time_us":24637,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":487,"num_input_rowsets":2,"peak_mem_usage":6400984,"rows_written":399450,"spinlock_wait_cycles":2816,"thread_start_us":185,"threads_started":5}
I20250113 03:10:47.639417 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4492150 bytes on disk
I20250113 03:10:47.639748 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.639865 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.665154 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.025s user 0.019s sys 0.000s Metrics: {"bytes_written":508388,"cfile_init":1,"dirs.queue_time_us":22,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17800}
I20250113 03:10:47.665377 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010617
I20250113 03:10:47.765543 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.440s user 0.321s sys 0.027s Metrics: {"bytes_written":11254200,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1055286,"cfile_cache_miss":494,"cfile_cache_miss_bytes":28565168,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":16023,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":512,"lbm_write_time_us":24875,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":497,"num_input_rowsets":2,"peak_mem_usage":6481377,"rows_written":404250,"spinlock_wait_cycles":384}
I20250113 03:10:47.765848 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4548478 bytes on disk
I20250113 03:10:47.767079 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.767423 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.794827 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.027s user 0.019s sys 0.000s Metrics: {"bytes_written":482889,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":525,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1973,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16950}
I20250113 03:10:47.795048 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 159965 bytes on disk
I20250113 03:10:47.795367 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.795706 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010575
I20250113 03:10:47.892621 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.404s user 0.324s sys 0.034s Metrics: {"bytes_written":11442493,"cfile_cache_hit":12,"cfile_cache_hit_bytes":736792,"cfile_cache_miss":506,"cfile_cache_miss_bytes":29371542,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":417,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":617,"drs_written":1,"lbm_read_time_us":12925,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":533,"lbm_write_time_us":18356,"lbm_writes_lt_1ms":505,"num_input_rowsets":2,"peak_mem_usage":6589858,"rows_written":411000,"spinlock_wait_cycles":1280}
I20250113 03:10:47.893533 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4626356 bytes on disk
I20250113 03:10:47.894529 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.894770 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:47.920055 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.017s sys 0.000s Metrics: {"bytes_written":414723,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2448,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":14400}
I20250113 03:10:47.920255 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1056817 bytes of WAL
I20250113 03:10:47.920370 31046 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:47.920440 31046 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 992c4345dfd4499390689246ae3d3cd9: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-1-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000014 (ops 7660-8241)
I20250113 03:10:47.920928 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:47.921053 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 135058 bytes on disk
I20250113 03:10:47.921759 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:47.921896 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010529
I20250113 03:10:48.055040 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.390s user 0.319s sys 0.036s Metrics: {"bytes_written":11615003,"cfile_cache_hit":11,"cfile_cache_hit_bytes":583150,"cfile_cache_miss":513,"cfile_cache_miss_bytes":29980484,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":4716,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":522,"drs_written":1,"lbm_read_time_us":13646,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":536,"lbm_write_time_us":17107,"lbm_writes_lt_1ms":513,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":6689059,"rows_written":417250,"spinlock_wait_cycles":1974144}
I20250113 03:10:48.055366 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4698903 bytes on disk
I20250113 03:10:48.055554 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20250113 03:10:48.055696 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.076474 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.012s sys 0.003s Metrics: {"bytes_written":398158,"cfile_init":1,"dirs.queue_time_us":20,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":817,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":13750}
I20250113 03:10:48.076812 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010438
I20250113 03:10:48.260756 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.465s user 0.317s sys 0.047s Metrics: {"bytes_written":11722936,"cfile_cache_hit":12,"cfile_cache_hit_bytes":740786,"cfile_cache_miss":518,"cfile_cache_miss_bytes":30113742,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":637,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":835,"drs_written":1,"lbm_read_time_us":12897,"lbm_reads_lt_1ms":538,"lbm_write_time_us":24389,"lbm_writes_lt_1ms":517,"num_input_rowsets":2,"peak_mem_usage":6754098,"rows_written":421200}
I20250113 03:10:48.261083 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.285193 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.024s user 0.010s sys 0.008s Metrics: {"bytes_written":462742,"cfile_init":1,"dirs.queue_time_us":264,"drs_written":1,"lbm_read_time_us":29,"lbm_reads_lt_1ms":4,"lbm_write_time_us":947,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16150}
I20250113 03:10:48.285435 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1055300 bytes of WAL
I20250113 03:10:48.285619 30970 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:48.285704 30970 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 80237f8c3f2c432facd353586e5b82d8: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-0-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000014 (ops 8046-8663)
I20250113 03:10:48.286101 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:48.286258 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010316
I20250113 03:10:48.305809 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.383s user 0.335s sys 0.024s Metrics: {"bytes_written":11838722,"cfile_cache_hit":10,"cfile_cache_hit_bytes":529590,"cfile_cache_miss":524,"cfile_cache_miss_bytes":30646448,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":876,"drs_written":1,"lbm_read_time_us":11402,"lbm_reads_lt_1ms":544,"lbm_write_time_us":18733,"lbm_writes_lt_1ms":521,"num_input_rowsets":2,"peak_mem_usage":6824892,"rows_written":425400,"spinlock_wait_cycles":5376}
I20250113 03:10:48.306008 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.323292 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":379507,"cfile_init":1,"dirs.queue_time_us":96,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1237,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13200}
I20250113 03:10:48.323480 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4917948 bytes on disk
I20250113 03:10:48.323745 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":137,"lbm_reads_lt_1ms":8}
I20250113 03:10:48.323945 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010170
I20250113 03:10:48.534534 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.458s user 0.352s sys 0.030s Metrics: {"bytes_written":11996004,"cfile_cache_hit":13,"cfile_cache_hit_bytes":890420,"cfile_cache_miss":527,"cfile_cache_miss_bytes":30697818,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":698,"dirs.run_wall_time_us":5326,"drs_written":1,"lbm_read_time_us":14634,"lbm_reads_lt_1ms":551,"lbm_write_time_us":24716,"lbm_writes_lt_1ms":529,"num_input_rowsets":2,"peak_mem_usage":6916534,"rows_written":431000,"spinlock_wait_cycles":768}
I20250113 03:10:48.534864 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4857164 bytes on disk
I20250113 03:10:48.535020 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20250113 03:10:48.535279 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.562057 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.027s user 0.014s sys 0.004s Metrics: {"bytes_written":413356,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":63,"dirs.run_wall_time_us":2056,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1282,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14350}
I20250113 03:10:48.562315 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): free 1053042 bytes of WAL
I20250113 03:10:48.562446 31121 log_reader.cc:385] T bee32ae0dcbc4912aeb5d6155f3158bb: removed 1 log segments from log reader
I20250113 03:10:48.562531 31121 log.cc:1078] T bee32ae0dcbc4912aeb5d6155f3158bb P 8e6ae3b64a874690b4c926cbede85053: Deleting log segment in path: /tmp/dist-test-taskbOgndE/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1736737556718722-20553-0/minicluster-data/ts-2-root/wals/bee32ae0dcbc4912aeb5d6155f3158bb/wal-000000014 (ops 7989-8629)
I20250113 03:10:48.563042 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: LogGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250113 03:10:48.563169 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 134595 bytes on disk
I20250113 03:10:48.563318 31121 maintenance_manager.cc:642] P 8e6ae3b64a874690b4c926cbede85053: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20250113 03:10:48.563560 31181 maintenance_manager.cc:418] P 8e6ae3b64a874690b4c926cbede85053: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010057
I20250113 03:10:48.773669 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.487s user 0.344s sys 0.040s Metrics: {"bytes_written":12168134,"cfile_cache_hit":11,"cfile_cache_hit_bytes":578614,"cfile_cache_miss":537,"cfile_cache_miss_bytes":31472450,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":575,"drs_written":1,"lbm_read_time_us":17917,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":563,"lbm_write_time_us":22459,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":534,"num_input_rowsets":2,"peak_mem_usage":7017315,"rows_written":437350,"spinlock_wait_cycles":1920,"thread_start_us":98,"threads_started":3}
I20250113 03:10:48.774202 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.776407 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.452s user 0.358s sys 0.021s Metrics: {"bytes_written":12205525,"cfile_cache_hit":13,"cfile_cache_hit_bytes":877118,"cfile_cache_miss":535,"cfile_cache_miss_bytes":31276326,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":267,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":1009,"drs_written":1,"lbm_read_time_us":16186,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":554,"lbm_write_time_us":20322,"lbm_writes_lt_1ms":537,"num_input_rowsets":2,"peak_mem_usage":7040580,"rows_written":438600,"spinlock_wait_cycles":1536}
I20250113 03:10:48.776572 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 4945981 bytes on disk
I20250113 03:10:48.776764 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20250113 03:10:48.776880 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.000000
I20250113 03:10:48.795188 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.021s user 0.014s sys 0.003s Metrics: {"bytes_written":455362,"cfile_init":1,"dirs.queue_time_us":20,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15850}
I20250113 03:10:48.795388 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 5080544 bytes on disk
I20250113 03:10:48.795584 30970 maintenance_manager.cc:642] P 80237f8c3f2c432facd353586e5b82d8: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20250113 03:10:48.795758 31030 maintenance_manager.cc:418] P 80237f8c3f2c432facd353586e5b82d8: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010169
I20250113 03:10:48.798723 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: FlushMRSOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":421001,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1398,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14650}
I20250113 03:10:48.798892 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb): 137459 bytes on disk
I20250113 03:10:48.799041 31046 maintenance_manager.cc:642] P 992c4345dfd4499390689246ae3d3cd9: UndoDeltaBlockGCOp(bee32ae0dcbc4912aeb5d6155f3158bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20250113 03:10:48.799165 31106 maintenance_manager.cc:418] P 992c4345dfd4499390689246ae3d3cd9: Scheduling CompactRowSetsOp(bee32ae0dcbc4912aeb5d6155f3158bb): perf score=1.010167
F20250113 03:10:48.806998 31344 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
*** Aborted at 1736737848 (unix time) try "date -d @1736737848" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005049) received by PID 20553 (TID 0x7fcfde3e9700) from PID 20553; stack trace: ***
F20250113 03:10:48.817138 31345 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
@ 0x7fd07902a980 (unknown) at ??:0
@ 0x7fd0772f5fb7 gsignal at ??:0
@ 0x7fd0772f7921 abort at ??:0
@ 0x7fd078162dcd google::LogMessage::Fail() at ??:0
@ 0x7fd078166b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd0781627cc google::LogMessage::Flush() at ??:0
@ 0x7fd078163f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd079978958 kudu::TestWorkload::GetNumberOfErrors() at ??:0
@ 0x7fd07997a40c kudu::TestWorkload::WriteThread() at ??:0
@ 0x7fd07797d6df (unknown) at ??:0
@ 0x7fd07901f6db start_thread at ??:0
@ 0x7fd0773d871f clone at ??:0